The context: Why is this problem important to solve?
The objectives: What is the intended goal?
The key questions: What are the key questions that need to be answered?
The problem formulation: What are we trying to solve using data science?
The data set consists of 3 folders, i.e., 'test', 'train', and 'validation'. Each of these folders has four subfolders:
‘happy’: Images of people who have happy facial expressions.
‘sad’: Images of people with sad or upset facial expressions.
‘surprise’: Images of people who have shocked or surprised facial expressions.
‘neutral’: Images of people showing no prominent emotion in their facial expression at all.
NOTE: Please use Google Colab from your browser for this notebook. Google.colab is NOT a library that can be downloaded locally on your device.
# Mount your drive
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
!pip install tensorflow tensorflow-addons efficientnet
Requirement already satisfied: tensorflow in /usr/local/lib/python3.10/dist-packages (2.15.0)
Collecting tensorflow-addons
Downloading tensorflow_addons-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (611 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 611.8/611.8 kB 9.8 MB/s eta 0:00:00
Collecting efficientnet
Downloading efficientnet-1.1.1-py3-none-any.whl (18 kB)
Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.4.0)
Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.6.3)
Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.5.26)
Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.5.4)
Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.9.0)
Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (16.0.6)
Requirement already satisfied: ml-dtypes~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)
Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.25.2)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.3.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow) (23.2)
Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.20.3)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow) (67.7.2)
Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.4.0)
Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (4.9.0)
Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.14.1)
Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.36.0)
Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.60.1)
Requirement already satisfied: tensorboard<2.16,>=2.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.2)
Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)
Requirement already satisfied: keras<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)
Collecting typeguard<3.0.0,>=2.7 (from tensorflow-addons)
Downloading typeguard-2.13.3-py3-none-any.whl (17 kB)
Collecting keras-applications<=1.0.8,>=1.0.7 (from efficientnet)
Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.7/50.7 kB 7.7 MB/s eta 0:00:00
Requirement already satisfied: scikit-image in /usr/local/lib/python3.10/dist-packages (from efficientnet) (0.19.3)
Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow) (0.42.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.27.0)
Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (1.2.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.5.2)
Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.31.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.0.1)
Requirement already satisfied: scipy>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image->efficientnet) (1.11.4)
Requirement already satisfied: networkx>=2.2 in /usr/local/lib/python3.10/dist-packages (from scikit-image->efficientnet) (3.2.1)
Requirement already satisfied: pillow!=7.1.0,!=7.1.1,!=8.3.0,>=6.1.0 in /usr/local/lib/python3.10/dist-packages (from scikit-image->efficientnet) (9.4.0)
Requirement already satisfied: imageio>=2.4.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image->efficientnet) (2.31.6)
Requirement already satisfied: tifffile>=2019.7.26 in /usr/local/lib/python3.10/dist-packages (from scikit-image->efficientnet) (2024.2.12)
Requirement already satisfied: PyWavelets>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-image->efficientnet) (1.5.0)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (1.3.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2024.2.2)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow) (2.1.5)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.5.1)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (3.2.2)
Installing collected packages: typeguard, tensorflow-addons, keras-applications, efficientnet
Successfully installed efficientnet-1.1.1 keras-applications-1.0.8 tensorflow-addons-0.23.0 typeguard-2.13.3
import numpy as np
import pandas as pd
import os
import zipfile
# For Data Visualization
import cv2
import seaborn as sns
import matplotlib.pyplot as plt
# For Model Building
import tensorflow as tf
import keras
from tensorflow.keras.utils import load_img, img_to_array
from keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.models import Sequential, Model # Sequential API for sequential model
from tensorflow.keras.layers import Dense, Dropout, Flatten # Importing different layers
from tensorflow.keras.layers import Conv2D, MaxPooling2D, BatchNormalization, Activation, Input, LeakyReLU
from tensorflow.keras import backend
from tensorflow.keras.utils import to_categorical # To perform one-hot encoding
from tensorflow.keras.optimizers import RMSprop, Adam, SGD # Optimizers for optimizing the model
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau
from tensorflow.keras import losses
from sklearn.metrics import confusion_matrix
from keras.applications import VGG16, ResNet50V2 # Regularization method to prevent the overfitting
from efficientnet.tfkeras import EfficientNetB0 # Regularization method to prevent the overfitting
Note:
# Storing the path of the data file from the Google drive
path = '/content/drive/MyDrive/face_emotion_recognition/Facial_emotion_images.zip'
# The data is provided as a zip file so we need to extract the files from the zip file
with zipfile.ZipFile(path, 'r') as zip_ref:
zip_ref.extractall()
Let's look at our classes.
Write down your observation for each class. What do you think can be a unique feature of each emotion, that separates it from the remaining classes?
folder_path = '/content/Facial_emotion_images/'
pic_size = 48
def file_number(folder_path):
# Initialize counter for files
files_number = 0
# Iterate through all the items in the folder
for _, _, files in os.walk(folder_path):
# Increment the counter for each file found
files_number += len(files)
return files_number
# Visualizing happy faces
emotion = 'happy'
plt.figure(figsize = (8, 8))
emotion_folder_path = (folder_path + "train/" + emotion)
happy_len = file_number(emotion_folder_path)
for i in range(1, 26, 1):
plt.subplot(5, 5, i)
random_index = np.random.randint(0, happy_len)
img = load_img(folder_path + "train/" + emotion + "/" +
os.listdir(folder_path + "train/" + emotion)[random_index], target_size = (pic_size, pic_size))
plt.xticks([])
plt.yticks([])
plt.imshow(img)
plt.show()
Observations and Insights:
# Visualizing sad faces
emotion = 'sad'
plt.figure(figsize = (8, 8))
emotion_folder_path = (folder_path + "train/" + emotion)
sad_len = file_number(emotion_folder_path)
for i in range(1, 26, 1):
plt.subplot(5, 5, i)
random_index = np.random.randint(0, sad_len)
img = load_img(folder_path + "train/" + emotion + "/" +
os.listdir(folder_path + "train/" + emotion)[random_index], target_size = (pic_size, pic_size))
plt.xticks([])
plt.yticks([])
plt.imshow(img)
plt.show()
Observations and Insights:
# Visualizing neutral faces
emotion = 'neutral'
plt.figure(figsize = (8, 8))
emotion_folder_path = (folder_path + "train/" + emotion)
neutral_len = file_number(emotion_folder_path)
for i in range(1, 26, 1):
plt.subplot(5, 5, i)
random_index = np.random.randint(0, neutral_len)
img = load_img(folder_path + "train/" + emotion + "/" +
os.listdir(folder_path + "train/" + emotion)[random_index], target_size = (pic_size, pic_size))
plt.xticks([])
plt.yticks([])
plt.imshow(img)
plt.show()
Observations and Insights:
# Visualizing surprised faces
emotion = 'surprise'
plt.figure(figsize = (8, 8))
emotion_folder_path = (folder_path + "train/" + emotion)
surprised_len = file_number(emotion_folder_path)
for i in range(1, 26, 1):
plt.subplot(5, 5, i)
random_index = np.random.randint(0, surprised_len)
img = load_img(folder_path + "train/" + emotion + "/" +
os.listdir(folder_path + "train/" + emotion)[random_index], target_size = (pic_size, pic_size))
plt.xticks([])
plt.yticks([])
plt.imshow(img)
plt.show()
Observations and Insights:
# creating a barplot with the number of pictures in training data set for each class
lens = [happy_len, sad_len, neutral_len,surprised_len]
countplot = sns.barplot(x= ['happy','sad','neutral', 'surprised'], y = lens)
plt.title('Emotion Class Frequency in Training Dataset')
plt.ylabel('Count')
plt.xlabel('Emotion Class')
for i, val in enumerate(lens):
countplot.text(i, val / 2, str(val), ha='center')
Observations and Insights:
Think About It:
In this section, we are creating data loaders that we will use as inputs to our Neural Network.
You have two options for the color_mode. You can set it to color_mode = 'rgb' or color_mode = 'grayscale'. You will need to try out both and see for yourself which one gives better performance.
# All images have to be rescaled by 1/255.
train_datagen = ImageDataGenerator(rescale = 1.0/255.0)
validation_datagen = ImageDataGenerator(rescale = 1.0/255.0)
test_datagen = ImageDataGenerator(rescale = 1.0/255.)
# images are stored in a structured folder and paths need to be set up load images from those folders
train_dir = os.path.join(folder_path, 'train')
test_dir = os.path.join(folder_path, 'test')
val_dir = os.path.join(folder_path, 'validation')
# Flowing training images in batches of 20 using train_datagen generator grayscale
train_generator_gray = train_datagen.flow_from_directory(train_dir,
batch_size=64,
class_mode='categorical',
target_size=(48, 48),
color_mode = 'grayscale')
# Flowing testing images in batches of 20 using test_datagen generator grayscale
test_generator_gray = test_datagen.flow_from_directory(test_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (48, 48),
color_mode = 'grayscale')
# Flowing validation images in batches of 20 using test_datagen generator grayscale
validation_generator_gray = validation_datagen.flow_from_directory(val_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (48, 48),
color_mode = 'grayscale')
Found 15109 images belonging to 4 classes. Found 128 images belonging to 4 classes. Found 4977 images belonging to 4 classes.
# Flowing training images in batches of 20 using train_datagen generator rgb
train_generator_rgb = train_datagen.flow_from_directory(train_dir,
batch_size=64,
class_mode='categorical',
target_size=(48, 48),
color_mode = 'rgb')
# Flowing testing images in batches of 20 using test_datagen generator rgb
test_generator_rgb = test_datagen.flow_from_directory(test_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (48, 48),
color_mode = 'rgb')
# Flowing validation images in batches of 20 using test_datagen generator rgb
validation_generator_rgb = validation_datagen.flow_from_directory(val_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (48, 48),
color_mode = 'rgb')
Found 15109 images belonging to 4 classes. Found 128 images belonging to 4 classes. Found 4977 images belonging to 4 classes.
# visualizing batch images from train_generator_gray
images, labels = next(train_generator_gray)
fig, axes = plt.subplots(4, 5, figsize = (8, 8))
fig.set_size_inches(8, 8)
for (image, label, ax) in zip(images, labels, axes.flatten()):
ax.imshow(image)
if label[0] == 1:
ax.set_title('happy')
elif label[1] == 1:
ax.set_title('neutral')
elif label[2] == 1:
ax.set_title('sad')
elif label[3] == 1:
ax.set_title('surprised')
ax.axis('off')
# visualizing batch images from train_generator_rgb
images, labels = next(train_generator_rgb)
fig, axes = plt.subplots(4, 5, figsize = (8, 8))
fig.set_size_inches(8, 8)
for (image, label, ax) in zip(images, labels, axes.flatten()):
ax.imshow(image)
if label[0] == 1:
ax.set_title('happy')
elif label[1] == 1:
ax.set_title('neutral')
elif label[2] == 1:
ax.set_title('sad')
elif label[3] == 1:
ax.set_title('surprised')
ax.axis('off')
# create a function to print classification report and draw heatmap for confusion matrix
def metrics_score(actual, predicted):
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
print(classification_report(actual, predicted))
cm = confusion_matrix(actual, predicted)
plt.figure(figsize = (8, 5))
sns.heatmap(cm, annot = True, fmt = '.0f', xticklabels = ['happy','neutral','sad','surprised'], yticklabels = ['happy','neutral','sad','surprised'])
plt.ylabel('Actual')
plt.xlabel('Predicted')
plt.show()
# Function to plot accuracy, val_accuracy,
def plot_accuracy(history):
N = len(history.history["accuracy"])
plt.figure()
plt.plot(np.arange(0, N), history.history["accuracy"], label="train_accuracy")
plt.plot(np.arange(0, N), history.history["val_accuracy"], label="val_accuracy")
plt.title("Training accuracy Dataset")
plt.xlabel("Epoch #")
plt.ylabel("accuracy")
plt.legend(loc="upper right")
# Function to plot loss, val_loss,
def plot_loss(history):
N = len(history.history["loss"])
plt.figure()
plt.plot(np.arange(0, N), history.history["loss"], label="train_loss")
plt.plot(np.arange(0, N), history.history["val_loss"], label="val_loss")
plt.title("Training loss Dataset")
plt.xlabel("Epoch #")
plt.ylabel("loss")
plt.legend(loc="upper right")
Think About It:
# clear any previous model's history from the Keras backend
from tensorflow.keras import backend
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(666)
import random
random.seed(666)
tf.random.set_seed(666)
# cnn for grayscale data with input shape of (48, 48, 1) for one color chanel
# model with 4 convolutional layers with leaky relu activation and added pooling and batch normalization
# fully connected part with 3 dense layers with added dropout layers for regularization
cnn_model_1 = Sequential()
cnn_model_1.add(Conv2D(64, (3,3), input_shape=(48, 48, 1), padding = 'same'))
cnn_model_1.add(LeakyReLU(0.1))
cnn_model_1.add(MaxPooling2D(2,2))
cnn_model_1.add(BatchNormalization())
cnn_model_1.add(Conv2D(32, (3,3), padding = 'same'))
cnn_model_1.add(LeakyReLU(0.2))
cnn_model_1.add(Conv2D(32, (3,3), padding = 'same'))
cnn_model_1.add(LeakyReLU(0.1))
cnn_model_1.add(MaxPooling2D(2,2))
cnn_model_1.add(BatchNormalization())
cnn_model_1.add(Conv2D(16, (3,3), padding = 'same'))
cnn_model_1.add(LeakyReLU(0.1))
cnn_model_1.add(MaxPooling2D(2,2))
cnn_model_1.add(BatchNormalization())
cnn_model_1.add(Flatten())
cnn_model_1.add(Dense(64, activation='relu'))
cnn_model_1.add(Dropout(0.35))
cnn_model_1.add(Dense(16, activation='relu'))
cnn_model_1.add(Dropout(0.2))
cnn_model_1.add(Dense(16, activation='relu'))
cnn_model_1.add(Dense(4, activation='softmax'))
# compiling the model with Adam optimizer, categorical crossentropy loss function as this is multi class model
# and accuracy metrics to evaluate the models performance during traiing and validation
opt = Adam(learning_rate = 0.001) #use Adam optimizer with default learning rate
cnn_model_1.compile(loss="categorical_crossentropy", optimizer=opt, metrics = ['accuracy'])
cnn_model_1.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_4 (Conv2D) (None, 48, 48, 64) 640
leaky_re_lu_4 (LeakyReLU) (None, 48, 48, 64) 0
max_pooling2d_3 (MaxPoolin (None, 24, 24, 64) 0
g2D)
batch_normalization_3 (Bat (None, 24, 24, 64) 256
chNormalization)
conv2d_5 (Conv2D) (None, 24, 24, 32) 18464
leaky_re_lu_5 (LeakyReLU) (None, 24, 24, 32) 0
conv2d_6 (Conv2D) (None, 24, 24, 32) 9248
leaky_re_lu_6 (LeakyReLU) (None, 24, 24, 32) 0
max_pooling2d_4 (MaxPoolin (None, 12, 12, 32) 0
g2D)
batch_normalization_4 (Bat (None, 12, 12, 32) 128
chNormalization)
conv2d_7 (Conv2D) (None, 12, 12, 16) 4624
leaky_re_lu_7 (LeakyReLU) (None, 12, 12, 16) 0
max_pooling2d_5 (MaxPoolin (None, 6, 6, 16) 0
g2D)
batch_normalization_5 (Bat (None, 6, 6, 16) 64
chNormalization)
flatten_1 (Flatten) (None, 576) 0
dense_4 (Dense) (None, 64) 36928
dropout_2 (Dropout) (None, 64) 0
dense_5 (Dense) (None, 16) 1040
dropout_3 (Dropout) (None, 16) 0
dense_6 (Dense) (None, 16) 272
dense_7 (Dense) (None, 4) 68
=================================================================
Total params: 71732 (280.20 KB)
Trainable params: 71508 (279.33 KB)
Non-trainable params: 224 (896.00 Byte)
_________________________________________________________________
# Pulling a single large batch of random validation data for validating after each epoch
valX, valY = validation_generator_gray.next()
# training the model of training data set and validating on validation data set
# training history saved to cnn_1_hist_gray
# Reduce LR on Plateau callback will reduce the learning rate to 0.2 of its current value, if val_loss does not improve for 5 consecutive epochs
learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_loss',
patience = 5,
verbose = 1,
factor = 0.2,
cooldown = 2,
min_lr = 0.0001)
cnn_1_hist_gray = cnn_model_1.fit(train_generator_gray,
validation_data=(valX, valY),
epochs=15,
callbacks = [learning_rate_reduction] )
Epoch 1/15 237/237 [==============================] - 9s 26ms/step - loss: 1.3022 - accuracy: 0.3712 - val_loss: 1.3064 - val_accuracy: 0.3672 - lr: 0.0010 Epoch 2/15 237/237 [==============================] - 6s 24ms/step - loss: 1.1190 - accuracy: 0.4963 - val_loss: 1.0593 - val_accuracy: 0.5156 - lr: 0.0010 Epoch 3/15 237/237 [==============================] - 6s 27ms/step - loss: 1.0137 - accuracy: 0.5564 - val_loss: 1.0276 - val_accuracy: 0.5781 - lr: 0.0010 Epoch 4/15 237/237 [==============================] - 5s 23ms/step - loss: 0.9433 - accuracy: 0.5920 - val_loss: 1.1184 - val_accuracy: 0.5547 - lr: 0.0010 Epoch 5/15 237/237 [==============================] - 6s 23ms/step - loss: 0.8802 - accuracy: 0.6263 - val_loss: 0.8266 - val_accuracy: 0.6953 - lr: 0.0010 Epoch 6/15 237/237 [==============================] - 7s 28ms/step - loss: 0.8212 - accuracy: 0.6595 - val_loss: 0.7643 - val_accuracy: 0.7109 - lr: 0.0010 Epoch 7/15 237/237 [==============================] - 5s 23ms/step - loss: 0.7840 - accuracy: 0.6730 - val_loss: 0.8155 - val_accuracy: 0.6797 - lr: 0.0010 Epoch 8/15 237/237 [==============================] - 6s 23ms/step - loss: 0.7484 - accuracy: 0.6871 - val_loss: 0.7266 - val_accuracy: 0.7422 - lr: 0.0010 Epoch 9/15 237/237 [==============================] - 6s 23ms/step - loss: 0.7203 - accuracy: 0.7021 - val_loss: 0.7824 - val_accuracy: 0.7188 - lr: 0.0010 Epoch 10/15 237/237 [==============================] - 7s 28ms/step - loss: 0.6816 - accuracy: 0.7222 - val_loss: 0.7352 - val_accuracy: 0.7578 - lr: 0.0010 Epoch 11/15 237/237 [==============================] - 6s 23ms/step - loss: 0.6676 - accuracy: 0.7265 - val_loss: 0.9389 - val_accuracy: 0.6641 - lr: 0.0010 Epoch 12/15 237/237 [==============================] - 6s 26ms/step - loss: 0.6314 - accuracy: 0.7439 - val_loss: 0.7398 - val_accuracy: 0.6641 - lr: 0.0010 Epoch 13/15 236/237 [============================>.] - ETA: 0s - loss: 0.6347 - accuracy: 0.7407 Epoch 13: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 237/237 [==============================] - 6s 26ms/step - loss: 0.6351 - accuracy: 0.7404 - val_loss: 0.7769 - val_accuracy: 0.7109 - lr: 0.0010 Epoch 14/15 237/237 [==============================] - 6s 24ms/step - loss: 0.5395 - accuracy: 0.7787 - val_loss: 0.7376 - val_accuracy: 0.7656 - lr: 2.0000e-04 Epoch 15/15 237/237 [==============================] - 7s 28ms/step - loss: 0.5106 - accuracy: 0.7923 - val_loss: 0.7548 - val_accuracy: 0.7656 - lr: 2.0000e-04
# evaluate the model on test data
cnn_model_1.evaluate(test_generator_gray)
1/1 [==============================] - 0s 269ms/step - loss: 0.6739 - accuracy: 0.7500
[0.6739221215248108, 0.75]
plot_accuracy(cnn_1_hist_gray)
plot_loss(cnn_1_hist_gray)
# get test images and their classes
test_images, test_classes = next(test_generator_gray)
actual_classes = np.argmax(test_classes, axis = -1)
# Make predictions
cnn_1_predictions = cnn_model_1.predict(test_images)
# Convert predictions to class labels
cnn_1_predictions = np.argmax(cnn_1_predictions, axis = -1)
# generate classification report and confussion matrix
metrics_score(actual_classes,cnn_1_predictions )
4/4 [==============================] - 0s 6ms/step
precision recall f1-score support
0 0.81 0.81 0.81 32
1 0.69 0.75 0.72 32
2 0.56 0.56 0.56 32
3 0.97 0.88 0.92 32
accuracy 0.75 128
macro avg 0.76 0.75 0.75 128
weighted avg 0.76 0.75 0.75 128
Observations and Insights:
# clear the previous model's history from the Keras backend
from tensorflow.keras import backend
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(666)
import random
random.seed(666)
tf.random.set_seed(666)
# running same model on rgb data to compare results
# cnn for rgb data with input shape of (48, 48, 3) for three color channels
# model with 4 convolutional layers with leaky relu activation and added pooling and batch normalization
# fully connected part with 3 dense layers with added dropout layers for regularization
cnn_model_1rgb = Sequential()
cnn_model_1rgb.add(Conv2D(64, (3,3), input_shape=(48, 48, 3), padding = 'same'))
cnn_model_1rgb.add(LeakyReLU(0.1))
cnn_model_1rgb.add(MaxPooling2D(2,2))
cnn_model_1rgb.add(BatchNormalization())
cnn_model_1rgb.add(Conv2D(32, (3,3), padding = 'same'))
cnn_model_1rgb.add(LeakyReLU(0.2))
cnn_model_1rgb.add(Conv2D(32, (3,3), padding = 'same'))
cnn_model_1rgb.add(LeakyReLU(0.1))
cnn_model_1rgb.add(MaxPooling2D(2,2))
cnn_model_1rgb.add(BatchNormalization())
cnn_model_1rgb.add(Conv2D(16, (3,3), padding = 'same'))
cnn_model_1rgb.add(LeakyReLU(0.1))
cnn_model_1rgb.add(MaxPooling2D(2,2))
cnn_model_1rgb.add(BatchNormalization())
cnn_model_1rgb.add(Flatten())
cnn_model_1rgb.add(Dense(64, activation='relu'))
cnn_model_1rgb.add(Dropout(0.35))
cnn_model_1rgb.add(Dense(16, activation='relu'))
cnn_model_1rgb.add(Dropout(0.2))
cnn_model_1rgb.add(Dense(16, activation='relu'))
cnn_model_1rgb.add(Dense(4, activation='softmax'))
# compiling the model with Adam optimizer, categorical crossentropy loss function as this is multi class model
# and accuracy metrics to evaluate the models performance during traiing and validation
opt = Adam(learning_rate = 0.001) #use Adam optimizer with default learning rate
cnn_model_1rgb.compile(loss="categorical_crossentropy", optimizer= opt, metrics = ['accuracy'])
cnn_model_1rgb.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 48, 48, 64) 1792
leaky_re_lu (LeakyReLU) (None, 48, 48, 64) 0
max_pooling2d (MaxPooling2 (None, 24, 24, 64) 0
D)
batch_normalization (Batch (None, 24, 24, 64) 256
Normalization)
conv2d_1 (Conv2D) (None, 24, 24, 32) 18464
leaky_re_lu_1 (LeakyReLU) (None, 24, 24, 32) 0
conv2d_2 (Conv2D) (None, 24, 24, 32) 9248
leaky_re_lu_2 (LeakyReLU) (None, 24, 24, 32) 0
max_pooling2d_1 (MaxPoolin (None, 12, 12, 32) 0
g2D)
batch_normalization_1 (Bat (None, 12, 12, 32) 128
chNormalization)
conv2d_3 (Conv2D) (None, 12, 12, 16) 4624
leaky_re_lu_3 (LeakyReLU) (None, 12, 12, 16) 0
max_pooling2d_2 (MaxPoolin (None, 6, 6, 16) 0
g2D)
batch_normalization_2 (Bat (None, 6, 6, 16) 64
chNormalization)
flatten (Flatten) (None, 576) 0
dense (Dense) (None, 64) 36928
dropout (Dropout) (None, 64) 0
dense_1 (Dense) (None, 16) 1040
dropout_1 (Dropout) (None, 16) 0
dense_2 (Dense) (None, 16) 272
dense_3 (Dense) (None, 4) 68
=================================================================
Total params: 72884 (284.70 KB)
Trainable params: 72660 (283.83 KB)
Non-trainable params: 224 (896.00 Byte)
_________________________________________________________________
# Pulling a single large batch of random validation data for validating after each epoch
valX, valY = validation_generator_rgb.next()
# training the model of training data set and validating on validation data set
# training history saved to cnn_1_hist_rgb
# Reduce LR on Plateau callback will reduce the learning rate to 0.2 of its current value, if val_loss does not improve for 5 consecutive epochs
learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_loss',
patience = 5,
verbose = 1,
factor = 0.2,
cooldown = 2,
min_lr = 0.0001)
cnn_1_hist_rgb = cnn_model_1rgb.fit(train_generator_rgb,
validation_data=(valX, valY),
epochs=15,
callbacks = [learning_rate_reduction] )
Epoch 1/15 237/237 [==============================] - 14s 31ms/step - loss: 1.3222 - accuracy: 0.3491 - val_loss: 1.2789 - val_accuracy: 0.4453 - lr: 0.0010 Epoch 2/15 237/237 [==============================] - 8s 33ms/step - loss: 1.1508 - accuracy: 0.4760 - val_loss: 1.2327 - val_accuracy: 0.4531 - lr: 0.0010 Epoch 3/15 237/237 [==============================] - 7s 31ms/step - loss: 1.0254 - accuracy: 0.5487 - val_loss: 1.0727 - val_accuracy: 0.5703 - lr: 0.0010 Epoch 4/15 237/237 [==============================] - 7s 29ms/step - loss: 0.9442 - accuracy: 0.5904 - val_loss: 1.4013 - val_accuracy: 0.3594 - lr: 0.0010 Epoch 5/15 237/237 [==============================] - 7s 31ms/step - loss: 0.8889 - accuracy: 0.6214 - val_loss: 0.9566 - val_accuracy: 0.5781 - lr: 0.0010 Epoch 6/15 237/237 [==============================] - 7s 31ms/step - loss: 0.8382 - accuracy: 0.6457 - val_loss: 0.9553 - val_accuracy: 0.6094 - lr: 0.0010 Epoch 7/15 237/237 [==============================] - 8s 33ms/step - loss: 0.8030 - accuracy: 0.6604 - val_loss: 0.8674 - val_accuracy: 0.6875 - lr: 0.0010 Epoch 8/15 237/237 [==============================] - 7s 29ms/step - loss: 0.7703 - accuracy: 0.6806 - val_loss: 0.8205 - val_accuracy: 0.6484 - lr: 0.0010 Epoch 9/15 237/237 [==============================] - 8s 33ms/step - loss: 0.7387 - accuracy: 0.6940 - val_loss: 0.8341 - val_accuracy: 0.6484 - lr: 0.0010 Epoch 10/15 237/237 [==============================] - 8s 33ms/step - loss: 0.7072 - accuracy: 0.7058 - val_loss: 0.8662 - val_accuracy: 0.6406 - lr: 0.0010 Epoch 11/15 237/237 [==============================] - 7s 29ms/step - loss: 0.6749 - accuracy: 0.7237 - val_loss: 1.0657 - val_accuracy: 0.5547 - lr: 0.0010 Epoch 12/15 237/237 [==============================] - 8s 36ms/step - loss: 0.6533 - accuracy: 0.7337 - val_loss: 0.8774 - val_accuracy: 0.6484 - lr: 0.0010 Epoch 13/15 237/237 [==============================] - ETA: 0s - loss: 0.6361 - accuracy: 0.7378 Epoch 13: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 237/237 [==============================] - 7s 30ms/step - loss: 0.6361 - accuracy: 0.7378 - val_loss: 0.8789 - val_accuracy: 0.6094 - lr: 0.0010 Epoch 14/15 237/237 [==============================] - 7s 28ms/step - loss: 0.5591 - accuracy: 0.7729 - val_loss: 0.8316 - val_accuracy: 0.6641 - lr: 2.0000e-04 Epoch 15/15 237/237 [==============================] - 8s 33ms/step - loss: 0.5271 - accuracy: 0.7869 - val_loss: 0.8282 - val_accuracy: 0.6484 - lr: 2.0000e-04
# evaluate the model on test data
cnn_model_1rgb.evaluate(test_generator_rgb)
1/1 [==============================] - 0s 422ms/step - loss: 0.6653 - accuracy: 0.7266
[0.6652733087539673, 0.7265625]
plot_accuracy(cnn_1_hist_rgb)
plot_loss(cnn_1_hist_rgb)
# get test images and their classes
test_images, test_classes = next(test_generator_rgb)
actual_classes = np.argmax(test_classes, axis = -1)
# Make predictions
cnn_1rgb_predictions = cnn_model_1rgb.predict(test_images)
# Convert predictions to class labels
cnn_1rgb_predictions = np.argmax(cnn_1rgb_predictions, axis = -1)
# generate classification report and confussion matrix
metrics_score(actual_classes,cnn_1rgb_predictions )
4/4 [==============================] - 0s 4ms/step
precision recall f1-score support
0 0.74 0.81 0.78 32
1 0.60 0.78 0.68 32
2 0.65 0.47 0.55 32
3 0.96 0.84 0.90 32
accuracy 0.73 128
macro avg 0.74 0.73 0.72 128
weighted avg 0.74 0.73 0.72 128
Observations and Insights:
# clear the previous model's history from the Keras backend
from tensorflow.keras import backend
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(666)
import random
random.seed(666)
tf.random.set_seed(666)
# building a bit larger architecture
cnn_model_2 = Sequential()
cnn_model_2.add(Conv2D(512, (3,3), input_shape=(48, 48, 3), padding = 'same'))
cnn_model_2.add(LeakyReLU(0.01))
cnn_model_2.add(BatchNormalization())
cnn_model_2.add(Conv2D(256, (3,3), padding = 'same'))
cnn_model_2.add(LeakyReLU(0.07))
cnn_model_2.add(Conv2D(256, (3,3), padding = 'same'))
cnn_model_2.add(LeakyReLU(0.1))
cnn_model_2.add(BatchNormalization())
cnn_model_2.add(MaxPooling2D(2,2))
cnn_model_2.add(Conv2D(64, (3,3), padding = 'same'))
cnn_model_2.add(LeakyReLU(0.1))
cnn_model_2.add(Conv2D(64, (3,3), padding = 'same'))
cnn_model_2.add(LeakyReLU(0.1))
cnn_model_2.add(BatchNormalization())
cnn_model_2.add(MaxPooling2D(2,2))
cnn_model_2.add(Conv2D(16, (3,3), padding = 'same'))
cnn_model_2.add(LeakyReLU(0.05))
cnn_model_2.add(BatchNormalization())
cnn_model_2.add(Flatten())
cnn_model_2.add(Dense(256))
cnn_model_2.add(LeakyReLU(0.1))
cnn_model_2.add(Dropout(0.25))
cnn_model_2.add(Dense(256))
cnn_model_2.add(LeakyReLU(0.1))
cnn_model_2.add(Dropout(0.25))
cnn_model_2.add(Dense(128))
cnn_model_2.add(LeakyReLU(0.1))
cnn_model_2.add(Dropout(0.25))
cnn_model_2.add(Dense(4, activation='softmax'))
# compiling the model with Adam optimizer, categorical crossentropy loss function as this is multi class model
# and accuracy metrics to evaluate the models performance during traiing and validation
opt = Adam(learning_rate = 0.001) #use Adam optimizer with default learning rate
cnn_model_2.compile(loss="categorical_crossentropy", optimizer="adam", metrics = ['accuracy'])
cnn_model_2.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 48, 48, 512) 14336
leaky_re_lu (LeakyReLU) (None, 48, 48, 512) 0
batch_normalization (Batch (None, 48, 48, 512) 2048
Normalization)
conv2d_1 (Conv2D) (None, 48, 48, 256) 1179904
leaky_re_lu_1 (LeakyReLU) (None, 48, 48, 256) 0
conv2d_2 (Conv2D) (None, 48, 48, 256) 590080
leaky_re_lu_2 (LeakyReLU) (None, 48, 48, 256) 0
batch_normalization_1 (Bat (None, 48, 48, 256) 1024
chNormalization)
max_pooling2d (MaxPooling2 (None, 24, 24, 256) 0
D)
conv2d_3 (Conv2D) (None, 24, 24, 64) 147520
leaky_re_lu_3 (LeakyReLU) (None, 24, 24, 64) 0
conv2d_4 (Conv2D) (None, 24, 24, 64) 36928
leaky_re_lu_4 (LeakyReLU) (None, 24, 24, 64) 0
batch_normalization_2 (Bat (None, 24, 24, 64) 256
chNormalization)
max_pooling2d_1 (MaxPoolin (None, 12, 12, 64) 0
g2D)
conv2d_5 (Conv2D) (None, 12, 12, 16) 9232
leaky_re_lu_5 (LeakyReLU) (None, 12, 12, 16) 0
batch_normalization_3 (Bat (None, 12, 12, 16) 64
chNormalization)
flatten (Flatten) (None, 2304) 0
dense (Dense) (None, 256) 590080
leaky_re_lu_6 (LeakyReLU) (None, 256) 0
dropout (Dropout) (None, 256) 0
dense_1 (Dense) (None, 256) 65792
leaky_re_lu_7 (LeakyReLU) (None, 256) 0
dropout_1 (Dropout) (None, 256) 0
dense_2 (Dense) (None, 128) 32896
leaky_re_lu_8 (LeakyReLU) (None, 128) 0
dropout_2 (Dropout) (None, 128) 0
dense_3 (Dense) (None, 4) 516
=================================================================
Total params: 2670676 (10.19 MB)
Trainable params: 2668980 (10.18 MB)
Non-trainable params: 1696 (6.62 KB)
_________________________________________________________________
# Pulling a single large batch of random validation data for validating after each epoch
valX, valY = validation_generator_rgb.next()
# the following code will stop the training if the val_loss continues above min val_loss for the run for more than 8 epochs
es = EarlyStopping(monitor = 'val_loss', mode = 'min', verbose = 1, patience = 8)
# Reduce LR on Plateau callback will reduce the learning rate to 0.2 of its current value, if val_loss does not improve for 5 consecutive epochs
learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_loss',
patience = 5,
verbose = 1,
factor = 0.2,
cooldown = 2,
min_lr = 0.0001)
# training the model of training data set and validating on validation data set
# training history saved to cnn_2_hist
cnn_2_hist = cnn_model_2.fit(train_generator_rgb,
validation_data=(valX, valY),
epochs=20,
callbacks = [es, learning_rate_reduction])
Epoch 1/20 237/237 [==============================] - 31s 79ms/step - loss: 1.2068 - accuracy: 0.4553 - val_loss: 1.2786 - val_accuracy: 0.3359 - lr: 0.0010 Epoch 2/20 237/237 [==============================] - 18s 77ms/step - loss: 0.9380 - accuracy: 0.5921 - val_loss: 0.9011 - val_accuracy: 0.5781 - lr: 0.0010 Epoch 3/20 237/237 [==============================] - 19s 79ms/step - loss: 0.8135 - accuracy: 0.6545 - val_loss: 0.8364 - val_accuracy: 0.6562 - lr: 0.0010 Epoch 4/20 237/237 [==============================] - 18s 76ms/step - loss: 0.7105 - accuracy: 0.7037 - val_loss: 0.7564 - val_accuracy: 0.6953 - lr: 0.0010 Epoch 5/20 237/237 [==============================] - 18s 76ms/step - loss: 0.6464 - accuracy: 0.7379 - val_loss: 0.8246 - val_accuracy: 0.6719 - lr: 0.0010 Epoch 6/20 237/237 [==============================] - 17s 70ms/step - loss: 0.5674 - accuracy: 0.7706 - val_loss: 0.6893 - val_accuracy: 0.6875 - lr: 0.0010 Epoch 7/20 237/237 [==============================] - 17s 71ms/step - loss: 0.4852 - accuracy: 0.8057 - val_loss: 0.7986 - val_accuracy: 0.6875 - lr: 0.0010 Epoch 8/20 237/237 [==============================] - 20s 83ms/step - loss: 0.4206 - accuracy: 0.8347 - val_loss: 0.7825 - val_accuracy: 0.6797 - lr: 0.0010 Epoch 9/20 237/237 [==============================] - 20s 83ms/step - loss: 0.3581 - accuracy: 0.8629 - val_loss: 0.9167 - val_accuracy: 0.6875 - lr: 0.0010 Epoch 10/20 237/237 [==============================] - 16s 69ms/step - loss: 0.3136 - accuracy: 0.8788 - val_loss: 0.8355 - val_accuracy: 0.7266 - lr: 0.0010 Epoch 11/20 237/237 [==============================] - ETA: 0s - loss: 0.2769 - accuracy: 0.8920 Epoch 11: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 237/237 [==============================] - 16s 69ms/step - loss: 0.2769 - accuracy: 0.8920 - val_loss: 0.9064 - val_accuracy: 0.7109 - lr: 0.0010 Epoch 12/20 237/237 [==============================] - 17s 69ms/step - loss: 0.1311 - accuracy: 0.9533 - val_loss: 0.8483 - val_accuracy: 0.7188 - lr: 2.0000e-04 Epoch 13/20 237/237 [==============================] - 16s 68ms/step - loss: 0.0826 - accuracy: 0.9709 - val_loss: 1.0050 - val_accuracy: 0.7422 - lr: 2.0000e-04 Epoch 14/20 237/237 [==============================] - 16s 69ms/step - loss: 0.0639 - accuracy: 0.9771 - val_loss: 1.1081 - val_accuracy: 0.7188 - lr: 2.0000e-04 Epoch 14: early stopping
# evaluate cnn_model_2 on test data
cnn_model_2.evaluate(test_generator_rgb)
1/1 [==============================] - 4s 4s/step - loss: 0.9150 - accuracy: 0.7812
[0.91496741771698, 0.78125]
plot_accuracy(cnn_2_hist)
plot_loss(cnn_2_hist)
# get test images and their classes
test_images, test_classes = next(test_generator_rgb)
actual_classes = np.argmax(test_classes, axis = -1)
# Make predictions
cnn_2_predictions = cnn_model_2.predict(test_images)
# Convert predictions to class labels
cnn_2_predictions = np.argmax(cnn_2_predictions, axis = -1)
# generate classification report and confussion matrix
metrics_score(actual_classes,cnn_2_predictions )
4/4 [==============================] - 0s 12ms/step
precision recall f1-score support
0 0.84 0.81 0.83 32
1 0.78 0.78 0.78 32
2 0.69 0.69 0.69 32
3 0.82 0.84 0.83 32
accuracy 0.78 128
macro avg 0.78 0.78 0.78 128
weighted avg 0.78 0.78 0.78 128
Observations and Insights:
In this section, we will create several Transfer Learning architectures. For the pre-trained models, we will select three popular architectures namely, VGG16, ResNet v2, and Efficient Net. The difference between these architectures and the previous architectures is that these will require 3 input channels while the earlier ones worked on 'grayscale' images. Therefore, we need to create new DataLoaders.
In this section, we are creating data loaders that we will use as inputs to our Neural Network. We will have to go with color_mode = 'rgb' as this is the required format for the transfer learning architectures.
# creating rbg loaders using the already initialized datagenerators
# this time changing target size to 224x224 to fit the pre tained networks' input shapes
# Flowing training images in batches of 20 using train_datagen generator rgb
train_generator_rgb_2 = train_datagen.flow_from_directory(train_dir,
batch_size=64,
class_mode='categorical',
target_size=(224, 224),
color_mode = 'rgb')
# Flowing testing images in batches of 20 using test_datagen generator rgb
test_generator_rgb_2 = test_datagen.flow_from_directory(test_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (224, 224),
color_mode = 'rgb')
# Flowing validation images in batches of 20 using test_datagen generator rgb
validation_generator_rgb_2 = validation_datagen.flow_from_directory(val_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (224, 224),
color_mode = 'rgb')
Found 15109 images belonging to 4 classes. Found 128 images belonging to 4 classes. Found 4977 images belonging to 4 classes.
# visualizing batch images from train_generator_rgb_2
images, labels = next(train_generator_rgb_2)
fig, axes = plt.subplots(4, 5, figsize = (8, 8))
fig.set_size_inches(8, 8)
for (image, label, ax) in zip(images, labels, axes.flatten()):
ax.imshow(image)
if label[0] == 1:
ax.set_title('happy')
elif label[1] == 1:
ax.set_title('neutral')
elif label[2] == 1:
ax.set_title('sad')
elif label[3] == 1:
ax.set_title('surprised')
ax.axis('off')
# clear the previous model's history from the Keras backend
from tensorflow.keras import backend
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(666)
import random
random.seed(666)
tf.random.set_seed(666)
# Summary of the whole VGG16 model
model = VGG16(weights='imagenet')
model.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels.h5
553467096/553467096 [==============================] - 16s 0us/step
Model: "vgg16"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0
block1_conv1 (Conv2D) (None, 224, 224, 64) 1792
block1_conv2 (Conv2D) (None, 224, 224, 64) 36928
block1_pool (MaxPooling2D) (None, 112, 112, 64) 0
block2_conv1 (Conv2D) (None, 112, 112, 128) 73856
block2_conv2 (Conv2D) (None, 112, 112, 128) 147584
block2_pool (MaxPooling2D) (None, 56, 56, 128) 0
block3_conv1 (Conv2D) (None, 56, 56, 256) 295168
block3_conv2 (Conv2D) (None, 56, 56, 256) 590080
block3_conv3 (Conv2D) (None, 56, 56, 256) 590080
block3_pool (MaxPooling2D) (None, 28, 28, 256) 0
block4_conv1 (Conv2D) (None, 28, 28, 512) 1180160
block4_conv2 (Conv2D) (None, 28, 28, 512) 2359808
block4_conv3 (Conv2D) (None, 28, 28, 512) 2359808
block4_pool (MaxPooling2D) (None, 14, 14, 512) 0
block5_conv1 (Conv2D) (None, 14, 14, 512) 2359808
block5_conv2 (Conv2D) (None, 14, 14, 512) 2359808
block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808
block5_pool (MaxPooling2D) (None, 7, 7, 512) 0
flatten (Flatten) (None, 25088) 0
fc1 (Dense) (None, 4096) 102764544
fc2 (Dense) (None, 4096) 16781312
predictions (Dense) (None, 1000) 4097000
=================================================================
Total params: 138357544 (527.79 MB)
Trainable params: 138357544 (527.79 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
# I will be using all 5 convoluitonal layers from VGG
transfer_layer = model.get_layer('block5_pool')
vgg_model = Model(inputs=model.input, outputs=transfer_layer.output)
vgg_model.summary()
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0
block1_conv1 (Conv2D) (None, 224, 224, 64) 1792
block1_conv2 (Conv2D) (None, 224, 224, 64) 36928
block1_pool (MaxPooling2D) (None, 112, 112, 64) 0
block2_conv1 (Conv2D) (None, 112, 112, 128) 73856
block2_conv2 (Conv2D) (None, 112, 112, 128) 147584
block2_pool (MaxPooling2D) (None, 56, 56, 128) 0
block3_conv1 (Conv2D) (None, 56, 56, 256) 295168
block3_conv2 (Conv2D) (None, 56, 56, 256) 590080
block3_conv3 (Conv2D) (None, 56, 56, 256) 590080
block3_pool (MaxPooling2D) (None, 28, 28, 256) 0
block4_conv1 (Conv2D) (None, 28, 28, 512) 1180160
block4_conv2 (Conv2D) (None, 28, 28, 512) 2359808
block4_conv3 (Conv2D) (None, 28, 28, 512) 2359808
block4_pool (MaxPooling2D) (None, 14, 14, 512) 0
block5_conv1 (Conv2D) (None, 14, 14, 512) 2359808
block5_conv2 (Conv2D) (None, 14, 14, 512) 2359808
block5_conv3 (Conv2D) (None, 14, 14, 512) 2359808
block5_pool (MaxPooling2D) (None, 7, 7, 512) 0
=================================================================
Total params: 14714688 (56.13 MB)
Trainable params: 14714688 (56.13 MB)
Non-trainable params: 0 (0.00 Byte)
_________________________________________________________________
# Making all the layers of the VGG model non-trainable. i.e. freezing them
for layer in vgg_model.layers:
layer.trainable = False
print(layer.name, layer.trainable)
input_1 False block1_conv1 False block1_conv2 False block1_pool False block2_conv1 False block2_conv2 False block2_pool False block3_conv1 False block3_conv2 False block3_conv3 False block3_pool False block4_conv1 False block4_conv2 False block4_conv3 False block4_pool False block5_conv1 False block5_conv2 False block5_conv3 False block5_pool False
# building CNN model with VGG engine
vgg_engine_model = Sequential()
# Adding the convolutional part of the VGG16 created above
vgg_engine_model.add(vgg_model)
# Flattening the output of the VGG16 engine
vgg_engine_model.add(Flatten())
# Adding a dense output layer
vgg_engine_model.add(Dense(32))
vgg_engine_model.add(LeakyReLU(0.1))
vgg_engine_model.add(Dropout(0.1))
vgg_engine_model.add(Dense(32))
vgg_engine_model.add(LeakyReLU(0.1))
vgg_engine_model.add(Dense(16))
vgg_engine_model.add(LeakyReLU(0.1))
vgg_engine_model.add(Dense(4, activation='softmax'))
# print model summary
vgg_engine_model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
model (Functional) (None, 7, 7, 512) 14714688
flatten (Flatten) (None, 25088) 0
dense (Dense) (None, 32) 802848
leaky_re_lu (LeakyReLU) (None, 32) 0
dropout (Dropout) (None, 32) 0
dense_1 (Dense) (None, 32) 1056
leaky_re_lu_1 (LeakyReLU) (None, 32) 0
dense_2 (Dense) (None, 16) 528
leaky_re_lu_2 (LeakyReLU) (None, 16) 0
dense_3 (Dense) (None, 4) 68
=================================================================
Total params: 15519188 (59.20 MB)
Trainable params: 804500 (3.07 MB)
Non-trainable params: 14714688 (56.13 MB)
_________________________________________________________________
# compiling the model with Adam optimizer, categorical crossentropy loss function as this is multi class model
# and accuracy metrics to evaluate the models performance during traiing and validation
opt = Adam(learning_rate = 0.001) #use Adam optimizer with default learning rate
vgg_engine_model.compile(loss="categorical_crossentropy", optimizer=opt, metrics = ['accuracy'])
# Pulling a single large batch of random validatio data for validating after each epoch
valX, valY = validation_generator_rgb_2.next()
# the following code will stop the training if the val_loss continues above min val_loss for the run for more than 8 epochs
es = EarlyStopping(monitor = 'val_loss', mode = 'min', verbose = 1, patience = 8)
# Reduce LR on Plateau callback will reduce the learning rate to 0.2 of its current value, if val_loss does not improve for 5 consecutive epochs
learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_loss',
patience = 5,
verbose = 1,
factor = 0.2,
cooldown = 2,
min_lr = 0.0001)
vgg_hist = vgg_engine_model.fit(train_generator_rgb_2,
validation_data=(valX, valY),
epochs=20,
callbacks = [es, learning_rate_reduction])
Epoch 1/20 237/237 [==============================] - 43s 160ms/step - loss: 1.1309 - accuracy: 0.4955 - val_loss: 0.9085 - val_accuracy: 0.6172 - lr: 0.0010 Epoch 2/20 237/237 [==============================] - 34s 141ms/step - loss: 0.8973 - accuracy: 0.6229 - val_loss: 0.8957 - val_accuracy: 0.5938 - lr: 0.0010 Epoch 3/20 237/237 [==============================] - 37s 158ms/step - loss: 0.8002 - accuracy: 0.6677 - val_loss: 0.8204 - val_accuracy: 0.6562 - lr: 0.0010 Epoch 4/20 237/237 [==============================] - 34s 141ms/step - loss: 0.7451 - accuracy: 0.6904 - val_loss: 0.9674 - val_accuracy: 0.5703 - lr: 0.0010 Epoch 5/20 237/237 [==============================] - 32s 136ms/step - loss: 0.7134 - accuracy: 0.7077 - val_loss: 0.8967 - val_accuracy: 0.6016 - lr: 0.0010 Epoch 6/20 237/237 [==============================] - 34s 143ms/step - loss: 0.6662 - accuracy: 0.7267 - val_loss: 1.3335 - val_accuracy: 0.5547 - lr: 0.0010 Epoch 7/20 237/237 [==============================] - 34s 141ms/step - loss: 0.6211 - accuracy: 0.7471 - val_loss: 0.9028 - val_accuracy: 0.6641 - lr: 0.0010 Epoch 8/20 237/237 [==============================] - ETA: 0s - loss: 0.5699 - accuracy: 0.7697 Epoch 8: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 237/237 [==============================] - 32s 137ms/step - loss: 0.5699 - accuracy: 0.7697 - val_loss: 1.0480 - val_accuracy: 0.6094 - lr: 0.0010 Epoch 9/20 237/237 [==============================] - 33s 140ms/step - loss: 0.4620 - accuracy: 0.8184 - val_loss: 0.8871 - val_accuracy: 0.6719 - lr: 2.0000e-04 Epoch 10/20 237/237 [==============================] - 34s 143ms/step - loss: 0.4321 - accuracy: 0.8314 - val_loss: 0.9507 - val_accuracy: 0.6641 - lr: 2.0000e-04 Epoch 11/20 237/237 [==============================] - 32s 133ms/step - loss: 0.4146 - accuracy: 0.8410 - val_loss: 0.9309 - val_accuracy: 0.6484 - lr: 2.0000e-04 Epoch 11: early stopping
# evaluate vgg_engine_model on test data
vgg_engine_model.evaluate(test_generator_rgb_2)
1/1 [==============================] - 9s 9s/step - loss: 0.7871 - accuracy: 0.6953
[0.7870687246322632, 0.6953125]
plot_accuracy(vgg_hist)
plot_loss(vgg_hist)
# get test images and their classes
test_images, test_classes = next(test_generator_rgb_2)
actual_classes = np.argmax(test_classes, axis = -1)
# Make predictions
vgg_predictions = vgg_engine_model.predict(test_images)
# Convert predictions to class labels
vgg_predictions = np.argmax(vgg_predictions, axis = -1)
# generate classification report and confussion matrix
metrics_score(actual_classes,vgg_predictions )
4/4 [==============================] - 0s 41ms/step
precision recall f1-score support
0 0.76 0.81 0.79 32
1 0.59 0.59 0.59 32
2 0.58 0.56 0.57 32
3 0.84 0.81 0.83 32
accuracy 0.70 128
macro avg 0.69 0.70 0.69 128
weighted avg 0.69 0.70 0.69 128
Think About It:
Observations and Insights:
Note: You can even go back and build your own architecture on top of the VGG16 Transfer layer and see if you can improve the performance
# clear the previous model's history from the Keras backend
from tensorflow.keras import backend
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(666)
import random
random.seed(666)
tf.random.set_seed(666)
model = ResNet50V2(weights='imagenet')
model.summary()
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50v2_weights_tf_dim_ordering_tf_kernels.h5
102869336/102869336 [==============================] - 3s 0us/step
Model: "resnet50v2"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0 []
conv1_pad (ZeroPadding2D) (None, 230, 230, 3) 0 ['input_1[0][0]']
conv1_conv (Conv2D) (None, 112, 112, 64) 9472 ['conv1_pad[0][0]']
pool1_pad (ZeroPadding2D) (None, 114, 114, 64) 0 ['conv1_conv[0][0]']
pool1_pool (MaxPooling2D) (None, 56, 56, 64) 0 ['pool1_pad[0][0]']
conv2_block1_preact_bn (Ba (None, 56, 56, 64) 256 ['pool1_pool[0][0]']
tchNormalization)
conv2_block1_preact_relu ( (None, 56, 56, 64) 0 ['conv2_block1_preact_bn[0][0]
Activation) ']
conv2_block1_1_conv (Conv2 (None, 56, 56, 64) 4096 ['conv2_block1_preact_relu[0][
D) 0]']
conv2_block1_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block1_1_conv[0][0]']
rmalization)
conv2_block1_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block1_1_bn[0][0]']
ation)
conv2_block1_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block1_1_relu[0][0]']
dding2D)
conv2_block1_2_conv (Conv2 (None, 56, 56, 64) 36864 ['conv2_block1_2_pad[0][0]']
D)
conv2_block1_2_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block1_2_conv[0][0]']
rmalization)
conv2_block1_2_relu (Activ (None, 56, 56, 64) 0 ['conv2_block1_2_bn[0][0]']
ation)
conv2_block1_0_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block1_preact_relu[0][
D) 0]']
conv2_block1_3_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block1_2_relu[0][0]']
D)
conv2_block1_out (Add) (None, 56, 56, 256) 0 ['conv2_block1_0_conv[0][0]',
'conv2_block1_3_conv[0][0]']
conv2_block2_preact_bn (Ba (None, 56, 56, 256) 1024 ['conv2_block1_out[0][0]']
tchNormalization)
conv2_block2_preact_relu ( (None, 56, 56, 256) 0 ['conv2_block2_preact_bn[0][0]
Activation) ']
conv2_block2_1_conv (Conv2 (None, 56, 56, 64) 16384 ['conv2_block2_preact_relu[0][
D) 0]']
conv2_block2_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block2_1_conv[0][0]']
rmalization)
conv2_block2_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block2_1_bn[0][0]']
ation)
conv2_block2_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block2_1_relu[0][0]']
dding2D)
conv2_block2_2_conv (Conv2 (None, 56, 56, 64) 36864 ['conv2_block2_2_pad[0][0]']
D)
conv2_block2_2_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block2_2_conv[0][0]']
rmalization)
conv2_block2_2_relu (Activ (None, 56, 56, 64) 0 ['conv2_block2_2_bn[0][0]']
ation)
conv2_block2_3_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block2_2_relu[0][0]']
D)
conv2_block2_out (Add) (None, 56, 56, 256) 0 ['conv2_block1_out[0][0]',
'conv2_block2_3_conv[0][0]']
conv2_block3_preact_bn (Ba (None, 56, 56, 256) 1024 ['conv2_block2_out[0][0]']
tchNormalization)
conv2_block3_preact_relu ( (None, 56, 56, 256) 0 ['conv2_block3_preact_bn[0][0]
Activation) ']
conv2_block3_1_conv (Conv2 (None, 56, 56, 64) 16384 ['conv2_block3_preact_relu[0][
D) 0]']
conv2_block3_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block3_1_conv[0][0]']
rmalization)
conv2_block3_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block3_1_bn[0][0]']
ation)
conv2_block3_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block3_1_relu[0][0]']
dding2D)
conv2_block3_2_conv (Conv2 (None, 28, 28, 64) 36864 ['conv2_block3_2_pad[0][0]']
D)
conv2_block3_2_bn (BatchNo (None, 28, 28, 64) 256 ['conv2_block3_2_conv[0][0]']
rmalization)
conv2_block3_2_relu (Activ (None, 28, 28, 64) 0 ['conv2_block3_2_bn[0][0]']
ation)
max_pooling2d (MaxPooling2 (None, 28, 28, 256) 0 ['conv2_block2_out[0][0]']
D)
conv2_block3_3_conv (Conv2 (None, 28, 28, 256) 16640 ['conv2_block3_2_relu[0][0]']
D)
conv2_block3_out (Add) (None, 28, 28, 256) 0 ['max_pooling2d[0][0]',
'conv2_block3_3_conv[0][0]']
conv3_block1_preact_bn (Ba (None, 28, 28, 256) 1024 ['conv2_block3_out[0][0]']
tchNormalization)
conv3_block1_preact_relu ( (None, 28, 28, 256) 0 ['conv3_block1_preact_bn[0][0]
Activation) ']
conv3_block1_1_conv (Conv2 (None, 28, 28, 128) 32768 ['conv3_block1_preact_relu[0][
D) 0]']
conv3_block1_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block1_1_conv[0][0]']
rmalization)
conv3_block1_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block1_1_bn[0][0]']
ation)
conv3_block1_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block1_1_relu[0][0]']
dding2D)
conv3_block1_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block1_2_pad[0][0]']
D)
conv3_block1_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block1_2_conv[0][0]']
rmalization)
conv3_block1_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block1_2_bn[0][0]']
ation)
conv3_block1_0_conv (Conv2 (None, 28, 28, 512) 131584 ['conv3_block1_preact_relu[0][
D) 0]']
conv3_block1_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block1_2_relu[0][0]']
D)
conv3_block1_out (Add) (None, 28, 28, 512) 0 ['conv3_block1_0_conv[0][0]',
'conv3_block1_3_conv[0][0]']
conv3_block2_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block1_out[0][0]']
tchNormalization)
conv3_block2_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block2_preact_bn[0][0]
Activation) ']
conv3_block2_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block2_preact_relu[0][
D) 0]']
conv3_block2_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block2_1_conv[0][0]']
rmalization)
conv3_block2_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block2_1_bn[0][0]']
ation)
conv3_block2_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block2_1_relu[0][0]']
dding2D)
conv3_block2_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block2_2_pad[0][0]']
D)
conv3_block2_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block2_2_conv[0][0]']
rmalization)
conv3_block2_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block2_2_bn[0][0]']
ation)
conv3_block2_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block2_2_relu[0][0]']
D)
conv3_block2_out (Add) (None, 28, 28, 512) 0 ['conv3_block1_out[0][0]',
'conv3_block2_3_conv[0][0]']
conv3_block3_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block2_out[0][0]']
tchNormalization)
conv3_block3_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block3_preact_bn[0][0]
Activation) ']
conv3_block3_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block3_preact_relu[0][
D) 0]']
conv3_block3_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block3_1_conv[0][0]']
rmalization)
conv3_block3_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block3_1_bn[0][0]']
ation)
conv3_block3_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block3_1_relu[0][0]']
dding2D)
conv3_block3_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block3_2_pad[0][0]']
D)
conv3_block3_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block3_2_conv[0][0]']
rmalization)
conv3_block3_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block3_2_bn[0][0]']
ation)
conv3_block3_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block3_2_relu[0][0]']
D)
conv3_block3_out (Add) (None, 28, 28, 512) 0 ['conv3_block2_out[0][0]',
'conv3_block3_3_conv[0][0]']
conv3_block4_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block3_out[0][0]']
tchNormalization)
conv3_block4_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block4_preact_bn[0][0]
Activation) ']
conv3_block4_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block4_preact_relu[0][
D) 0]']
conv3_block4_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block4_1_conv[0][0]']
rmalization)
conv3_block4_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block4_1_bn[0][0]']
ation)
conv3_block4_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block4_1_relu[0][0]']
dding2D)
conv3_block4_2_conv (Conv2 (None, 14, 14, 128) 147456 ['conv3_block4_2_pad[0][0]']
D)
conv3_block4_2_bn (BatchNo (None, 14, 14, 128) 512 ['conv3_block4_2_conv[0][0]']
rmalization)
conv3_block4_2_relu (Activ (None, 14, 14, 128) 0 ['conv3_block4_2_bn[0][0]']
ation)
max_pooling2d_1 (MaxPoolin (None, 14, 14, 512) 0 ['conv3_block3_out[0][0]']
g2D)
conv3_block4_3_conv (Conv2 (None, 14, 14, 512) 66048 ['conv3_block4_2_relu[0][0]']
D)
conv3_block4_out (Add) (None, 14, 14, 512) 0 ['max_pooling2d_1[0][0]',
'conv3_block4_3_conv[0][0]']
conv4_block1_preact_bn (Ba (None, 14, 14, 512) 2048 ['conv3_block4_out[0][0]']
tchNormalization)
conv4_block1_preact_relu ( (None, 14, 14, 512) 0 ['conv4_block1_preact_bn[0][0]
Activation) ']
conv4_block1_1_conv (Conv2 (None, 14, 14, 256) 131072 ['conv4_block1_preact_relu[0][
D) 0]']
conv4_block1_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block1_1_conv[0][0]']
rmalization)
conv4_block1_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block1_1_bn[0][0]']
ation)
conv4_block1_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block1_1_relu[0][0]']
dding2D)
conv4_block1_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block1_2_pad[0][0]']
D)
conv4_block1_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block1_2_conv[0][0]']
rmalization)
conv4_block1_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block1_2_bn[0][0]']
ation)
conv4_block1_0_conv (Conv2 (None, 14, 14, 1024) 525312 ['conv4_block1_preact_relu[0][
D) 0]']
conv4_block1_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block1_2_relu[0][0]']
D)
conv4_block1_out (Add) (None, 14, 14, 1024) 0 ['conv4_block1_0_conv[0][0]',
'conv4_block1_3_conv[0][0]']
conv4_block2_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block1_out[0][0]']
tchNormalization)
conv4_block2_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block2_preact_bn[0][0]
Activation) ']
conv4_block2_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block2_preact_relu[0][
D) 0]']
conv4_block2_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block2_1_conv[0][0]']
rmalization)
conv4_block2_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block2_1_bn[0][0]']
ation)
conv4_block2_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block2_1_relu[0][0]']
dding2D)
conv4_block2_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block2_2_pad[0][0]']
D)
conv4_block2_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block2_2_conv[0][0]']
rmalization)
conv4_block2_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block2_2_bn[0][0]']
ation)
conv4_block2_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block2_2_relu[0][0]']
D)
conv4_block2_out (Add) (None, 14, 14, 1024) 0 ['conv4_block1_out[0][0]',
'conv4_block2_3_conv[0][0]']
conv4_block3_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block2_out[0][0]']
tchNormalization)
conv4_block3_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block3_preact_bn[0][0]
Activation) ']
conv4_block3_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block3_preact_relu[0][
D) 0]']
conv4_block3_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block3_1_conv[0][0]']
rmalization)
conv4_block3_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block3_1_bn[0][0]']
ation)
conv4_block3_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block3_1_relu[0][0]']
dding2D)
conv4_block3_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block3_2_pad[0][0]']
D)
conv4_block3_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block3_2_conv[0][0]']
rmalization)
conv4_block3_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block3_2_bn[0][0]']
ation)
conv4_block3_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block3_2_relu[0][0]']
D)
conv4_block3_out (Add) (None, 14, 14, 1024) 0 ['conv4_block2_out[0][0]',
'conv4_block3_3_conv[0][0]']
conv4_block4_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block3_out[0][0]']
tchNormalization)
conv4_block4_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block4_preact_bn[0][0]
Activation) ']
conv4_block4_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block4_preact_relu[0][
D) 0]']
conv4_block4_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block4_1_conv[0][0]']
rmalization)
conv4_block4_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block4_1_bn[0][0]']
ation)
conv4_block4_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block4_1_relu[0][0]']
dding2D)
conv4_block4_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block4_2_pad[0][0]']
D)
conv4_block4_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block4_2_conv[0][0]']
rmalization)
conv4_block4_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block4_2_bn[0][0]']
ation)
conv4_block4_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block4_2_relu[0][0]']
D)
conv4_block4_out (Add) (None, 14, 14, 1024) 0 ['conv4_block3_out[0][0]',
'conv4_block4_3_conv[0][0]']
conv4_block5_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block4_out[0][0]']
tchNormalization)
conv4_block5_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block5_preact_bn[0][0]
Activation) ']
conv4_block5_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block5_preact_relu[0][
D) 0]']
conv4_block5_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block5_1_conv[0][0]']
rmalization)
conv4_block5_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block5_1_bn[0][0]']
ation)
conv4_block5_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block5_1_relu[0][0]']
dding2D)
conv4_block5_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block5_2_pad[0][0]']
D)
conv4_block5_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block5_2_conv[0][0]']
rmalization)
conv4_block5_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block5_2_bn[0][0]']
ation)
conv4_block5_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block5_2_relu[0][0]']
D)
conv4_block5_out (Add) (None, 14, 14, 1024) 0 ['conv4_block4_out[0][0]',
'conv4_block5_3_conv[0][0]']
conv4_block6_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block5_out[0][0]']
tchNormalization)
conv4_block6_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block6_preact_bn[0][0]
Activation) ']
conv4_block6_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block6_preact_relu[0][
D) 0]']
conv4_block6_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block6_1_conv[0][0]']
rmalization)
conv4_block6_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block6_1_bn[0][0]']
ation)
conv4_block6_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block6_1_relu[0][0]']
dding2D)
conv4_block6_2_conv (Conv2 (None, 7, 7, 256) 589824 ['conv4_block6_2_pad[0][0]']
D)
conv4_block6_2_bn (BatchNo (None, 7, 7, 256) 1024 ['conv4_block6_2_conv[0][0]']
rmalization)
conv4_block6_2_relu (Activ (None, 7, 7, 256) 0 ['conv4_block6_2_bn[0][0]']
ation)
max_pooling2d_2 (MaxPoolin (None, 7, 7, 1024) 0 ['conv4_block5_out[0][0]']
g2D)
conv4_block6_3_conv (Conv2 (None, 7, 7, 1024) 263168 ['conv4_block6_2_relu[0][0]']
D)
conv4_block6_out (Add) (None, 7, 7, 1024) 0 ['max_pooling2d_2[0][0]',
'conv4_block6_3_conv[0][0]']
conv5_block1_preact_bn (Ba (None, 7, 7, 1024) 4096 ['conv4_block6_out[0][0]']
tchNormalization)
conv5_block1_preact_relu ( (None, 7, 7, 1024) 0 ['conv5_block1_preact_bn[0][0]
Activation) ']
conv5_block1_1_conv (Conv2 (None, 7, 7, 512) 524288 ['conv5_block1_preact_relu[0][
D) 0]']
conv5_block1_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block1_1_conv[0][0]']
rmalization)
conv5_block1_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block1_1_bn[0][0]']
ation)
conv5_block1_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block1_1_relu[0][0]']
dding2D)
conv5_block1_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block1_2_pad[0][0]']
D)
conv5_block1_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block1_2_conv[0][0]']
rmalization)
conv5_block1_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block1_2_bn[0][0]']
ation)
conv5_block1_0_conv (Conv2 (None, 7, 7, 2048) 2099200 ['conv5_block1_preact_relu[0][
D) 0]']
conv5_block1_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block1_2_relu[0][0]']
D)
conv5_block1_out (Add) (None, 7, 7, 2048) 0 ['conv5_block1_0_conv[0][0]',
'conv5_block1_3_conv[0][0]']
conv5_block2_preact_bn (Ba (None, 7, 7, 2048) 8192 ['conv5_block1_out[0][0]']
tchNormalization)
conv5_block2_preact_relu ( (None, 7, 7, 2048) 0 ['conv5_block2_preact_bn[0][0]
Activation) ']
conv5_block2_1_conv (Conv2 (None, 7, 7, 512) 1048576 ['conv5_block2_preact_relu[0][
D) 0]']
conv5_block2_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block2_1_conv[0][0]']
rmalization)
conv5_block2_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block2_1_bn[0][0]']
ation)
conv5_block2_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block2_1_relu[0][0]']
dding2D)
conv5_block2_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block2_2_pad[0][0]']
D)
conv5_block2_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block2_2_conv[0][0]']
rmalization)
conv5_block2_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block2_2_bn[0][0]']
ation)
conv5_block2_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block2_2_relu[0][0]']
D)
conv5_block2_out (Add) (None, 7, 7, 2048) 0 ['conv5_block1_out[0][0]',
'conv5_block2_3_conv[0][0]']
conv5_block3_preact_bn (Ba (None, 7, 7, 2048) 8192 ['conv5_block2_out[0][0]']
tchNormalization)
conv5_block3_preact_relu ( (None, 7, 7, 2048) 0 ['conv5_block3_preact_bn[0][0]
Activation) ']
conv5_block3_1_conv (Conv2 (None, 7, 7, 512) 1048576 ['conv5_block3_preact_relu[0][
D) 0]']
conv5_block3_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block3_1_conv[0][0]']
rmalization)
conv5_block3_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block3_1_bn[0][0]']
ation)
conv5_block3_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block3_1_relu[0][0]']
dding2D)
conv5_block3_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block3_2_pad[0][0]']
D)
conv5_block3_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block3_2_conv[0][0]']
rmalization)
conv5_block3_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block3_2_bn[0][0]']
ation)
conv5_block3_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block3_2_relu[0][0]']
D)
conv5_block3_out (Add) (None, 7, 7, 2048) 0 ['conv5_block2_out[0][0]',
'conv5_block3_3_conv[0][0]']
post_bn (BatchNormalizatio (None, 7, 7, 2048) 8192 ['conv5_block3_out[0][0]']
n)
post_relu (Activation) (None, 7, 7, 2048) 0 ['post_bn[0][0]']
avg_pool (GlobalAveragePoo (None, 2048) 0 ['post_relu[0][0]']
ling2D)
predictions (Dense) (None, 1000) 2049000 ['avg_pool[0][0]']
==================================================================================================
Total params: 25613800 (97.71 MB)
Trainable params: 25568360 (97.54 MB)
Non-trainable params: 45440 (177.50 KB)
__________________________________________________________________________________________________
# again, I will use the entire Resnet v2 architechture up to avg_pool (GlobalAveragePooling2D)
transfer_layer = model.get_layer('avg_pool')
resnet_model = Model(inputs=model.input, outputs=transfer_layer.output)
resnet_model.summary()
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0 []
conv1_pad (ZeroPadding2D) (None, 230, 230, 3) 0 ['input_1[0][0]']
conv1_conv (Conv2D) (None, 112, 112, 64) 9472 ['conv1_pad[0][0]']
pool1_pad (ZeroPadding2D) (None, 114, 114, 64) 0 ['conv1_conv[0][0]']
pool1_pool (MaxPooling2D) (None, 56, 56, 64) 0 ['pool1_pad[0][0]']
conv2_block1_preact_bn (Ba (None, 56, 56, 64) 256 ['pool1_pool[0][0]']
tchNormalization)
conv2_block1_preact_relu ( (None, 56, 56, 64) 0 ['conv2_block1_preact_bn[0][0]
Activation) ']
conv2_block1_1_conv (Conv2 (None, 56, 56, 64) 4096 ['conv2_block1_preact_relu[0][
D) 0]']
conv2_block1_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block1_1_conv[0][0]']
rmalization)
conv2_block1_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block1_1_bn[0][0]']
ation)
conv2_block1_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block1_1_relu[0][0]']
dding2D)
conv2_block1_2_conv (Conv2 (None, 56, 56, 64) 36864 ['conv2_block1_2_pad[0][0]']
D)
conv2_block1_2_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block1_2_conv[0][0]']
rmalization)
conv2_block1_2_relu (Activ (None, 56, 56, 64) 0 ['conv2_block1_2_bn[0][0]']
ation)
conv2_block1_0_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block1_preact_relu[0][
D) 0]']
conv2_block1_3_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block1_2_relu[0][0]']
D)
conv2_block1_out (Add) (None, 56, 56, 256) 0 ['conv2_block1_0_conv[0][0]',
'conv2_block1_3_conv[0][0]']
conv2_block2_preact_bn (Ba (None, 56, 56, 256) 1024 ['conv2_block1_out[0][0]']
tchNormalization)
conv2_block2_preact_relu ( (None, 56, 56, 256) 0 ['conv2_block2_preact_bn[0][0]
Activation) ']
conv2_block2_1_conv (Conv2 (None, 56, 56, 64) 16384 ['conv2_block2_preact_relu[0][
D) 0]']
conv2_block2_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block2_1_conv[0][0]']
rmalization)
conv2_block2_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block2_1_bn[0][0]']
ation)
conv2_block2_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block2_1_relu[0][0]']
dding2D)
conv2_block2_2_conv (Conv2 (None, 56, 56, 64) 36864 ['conv2_block2_2_pad[0][0]']
D)
conv2_block2_2_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block2_2_conv[0][0]']
rmalization)
conv2_block2_2_relu (Activ (None, 56, 56, 64) 0 ['conv2_block2_2_bn[0][0]']
ation)
conv2_block2_3_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block2_2_relu[0][0]']
D)
conv2_block2_out (Add) (None, 56, 56, 256) 0 ['conv2_block1_out[0][0]',
'conv2_block2_3_conv[0][0]']
conv2_block3_preact_bn (Ba (None, 56, 56, 256) 1024 ['conv2_block2_out[0][0]']
tchNormalization)
conv2_block3_preact_relu ( (None, 56, 56, 256) 0 ['conv2_block3_preact_bn[0][0]
Activation) ']
conv2_block3_1_conv (Conv2 (None, 56, 56, 64) 16384 ['conv2_block3_preact_relu[0][
D) 0]']
conv2_block3_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block3_1_conv[0][0]']
rmalization)
conv2_block3_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block3_1_bn[0][0]']
ation)
conv2_block3_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block3_1_relu[0][0]']
dding2D)
conv2_block3_2_conv (Conv2 (None, 28, 28, 64) 36864 ['conv2_block3_2_pad[0][0]']
D)
conv2_block3_2_bn (BatchNo (None, 28, 28, 64) 256 ['conv2_block3_2_conv[0][0]']
rmalization)
conv2_block3_2_relu (Activ (None, 28, 28, 64) 0 ['conv2_block3_2_bn[0][0]']
ation)
max_pooling2d (MaxPooling2 (None, 28, 28, 256) 0 ['conv2_block2_out[0][0]']
D)
conv2_block3_3_conv (Conv2 (None, 28, 28, 256) 16640 ['conv2_block3_2_relu[0][0]']
D)
conv2_block3_out (Add) (None, 28, 28, 256) 0 ['max_pooling2d[0][0]',
'conv2_block3_3_conv[0][0]']
conv3_block1_preact_bn (Ba (None, 28, 28, 256) 1024 ['conv2_block3_out[0][0]']
tchNormalization)
conv3_block1_preact_relu ( (None, 28, 28, 256) 0 ['conv3_block1_preact_bn[0][0]
Activation) ']
conv3_block1_1_conv (Conv2 (None, 28, 28, 128) 32768 ['conv3_block1_preact_relu[0][
D) 0]']
conv3_block1_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block1_1_conv[0][0]']
rmalization)
conv3_block1_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block1_1_bn[0][0]']
ation)
conv3_block1_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block1_1_relu[0][0]']
dding2D)
conv3_block1_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block1_2_pad[0][0]']
D)
conv3_block1_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block1_2_conv[0][0]']
rmalization)
conv3_block1_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block1_2_bn[0][0]']
ation)
conv3_block1_0_conv (Conv2 (None, 28, 28, 512) 131584 ['conv3_block1_preact_relu[0][
D) 0]']
conv3_block1_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block1_2_relu[0][0]']
D)
conv3_block1_out (Add) (None, 28, 28, 512) 0 ['conv3_block1_0_conv[0][0]',
'conv3_block1_3_conv[0][0]']
conv3_block2_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block1_out[0][0]']
tchNormalization)
conv3_block2_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block2_preact_bn[0][0]
Activation) ']
conv3_block2_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block2_preact_relu[0][
D) 0]']
conv3_block2_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block2_1_conv[0][0]']
rmalization)
conv3_block2_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block2_1_bn[0][0]']
ation)
conv3_block2_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block2_1_relu[0][0]']
dding2D)
conv3_block2_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block2_2_pad[0][0]']
D)
conv3_block2_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block2_2_conv[0][0]']
rmalization)
conv3_block2_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block2_2_bn[0][0]']
ation)
conv3_block2_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block2_2_relu[0][0]']
D)
conv3_block2_out (Add) (None, 28, 28, 512) 0 ['conv3_block1_out[0][0]',
'conv3_block2_3_conv[0][0]']
conv3_block3_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block2_out[0][0]']
tchNormalization)
conv3_block3_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block3_preact_bn[0][0]
Activation) ']
conv3_block3_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block3_preact_relu[0][
D) 0]']
conv3_block3_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block3_1_conv[0][0]']
rmalization)
conv3_block3_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block3_1_bn[0][0]']
ation)
conv3_block3_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block3_1_relu[0][0]']
dding2D)
conv3_block3_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block3_2_pad[0][0]']
D)
conv3_block3_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block3_2_conv[0][0]']
rmalization)
conv3_block3_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block3_2_bn[0][0]']
ation)
conv3_block3_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block3_2_relu[0][0]']
D)
conv3_block3_out (Add) (None, 28, 28, 512) 0 ['conv3_block2_out[0][0]',
'conv3_block3_3_conv[0][0]']
conv3_block4_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block3_out[0][0]']
tchNormalization)
conv3_block4_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block4_preact_bn[0][0]
Activation) ']
conv3_block4_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block4_preact_relu[0][
D) 0]']
conv3_block4_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block4_1_conv[0][0]']
rmalization)
conv3_block4_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block4_1_bn[0][0]']
ation)
conv3_block4_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block4_1_relu[0][0]']
dding2D)
conv3_block4_2_conv (Conv2 (None, 14, 14, 128) 147456 ['conv3_block4_2_pad[0][0]']
D)
conv3_block4_2_bn (BatchNo (None, 14, 14, 128) 512 ['conv3_block4_2_conv[0][0]']
rmalization)
conv3_block4_2_relu (Activ (None, 14, 14, 128) 0 ['conv3_block4_2_bn[0][0]']
ation)
max_pooling2d_1 (MaxPoolin (None, 14, 14, 512) 0 ['conv3_block3_out[0][0]']
g2D)
conv3_block4_3_conv (Conv2 (None, 14, 14, 512) 66048 ['conv3_block4_2_relu[0][0]']
D)
conv3_block4_out (Add) (None, 14, 14, 512) 0 ['max_pooling2d_1[0][0]',
'conv3_block4_3_conv[0][0]']
conv4_block1_preact_bn (Ba (None, 14, 14, 512) 2048 ['conv3_block4_out[0][0]']
tchNormalization)
conv4_block1_preact_relu ( (None, 14, 14, 512) 0 ['conv4_block1_preact_bn[0][0]
Activation) ']
conv4_block1_1_conv (Conv2 (None, 14, 14, 256) 131072 ['conv4_block1_preact_relu[0][
D) 0]']
conv4_block1_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block1_1_conv[0][0]']
rmalization)
conv4_block1_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block1_1_bn[0][0]']
ation)
conv4_block1_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block1_1_relu[0][0]']
dding2D)
conv4_block1_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block1_2_pad[0][0]']
D)
conv4_block1_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block1_2_conv[0][0]']
rmalization)
conv4_block1_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block1_2_bn[0][0]']
ation)
conv4_block1_0_conv (Conv2 (None, 14, 14, 1024) 525312 ['conv4_block1_preact_relu[0][
D) 0]']
conv4_block1_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block1_2_relu[0][0]']
D)
conv4_block1_out (Add) (None, 14, 14, 1024) 0 ['conv4_block1_0_conv[0][0]',
'conv4_block1_3_conv[0][0]']
conv4_block2_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block1_out[0][0]']
tchNormalization)
conv4_block2_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block2_preact_bn[0][0]
Activation) ']
conv4_block2_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block2_preact_relu[0][
D) 0]']
conv4_block2_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block2_1_conv[0][0]']
rmalization)
conv4_block2_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block2_1_bn[0][0]']
ation)
conv4_block2_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block2_1_relu[0][0]']
dding2D)
conv4_block2_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block2_2_pad[0][0]']
D)
conv4_block2_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block2_2_conv[0][0]']
rmalization)
conv4_block2_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block2_2_bn[0][0]']
ation)
conv4_block2_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block2_2_relu[0][0]']
D)
conv4_block2_out (Add) (None, 14, 14, 1024) 0 ['conv4_block1_out[0][0]',
'conv4_block2_3_conv[0][0]']
conv4_block3_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block2_out[0][0]']
tchNormalization)
conv4_block3_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block3_preact_bn[0][0]
Activation) ']
conv4_block3_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block3_preact_relu[0][
D) 0]']
conv4_block3_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block3_1_conv[0][0]']
rmalization)
conv4_block3_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block3_1_bn[0][0]']
ation)
conv4_block3_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block3_1_relu[0][0]']
dding2D)
conv4_block3_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block3_2_pad[0][0]']
D)
conv4_block3_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block3_2_conv[0][0]']
rmalization)
conv4_block3_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block3_2_bn[0][0]']
ation)
conv4_block3_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block3_2_relu[0][0]']
D)
conv4_block3_out (Add) (None, 14, 14, 1024) 0 ['conv4_block2_out[0][0]',
'conv4_block3_3_conv[0][0]']
conv4_block4_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block3_out[0][0]']
tchNormalization)
conv4_block4_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block4_preact_bn[0][0]
Activation) ']
conv4_block4_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block4_preact_relu[0][
D) 0]']
conv4_block4_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block4_1_conv[0][0]']
rmalization)
conv4_block4_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block4_1_bn[0][0]']
ation)
conv4_block4_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block4_1_relu[0][0]']
dding2D)
conv4_block4_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block4_2_pad[0][0]']
D)
conv4_block4_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block4_2_conv[0][0]']
rmalization)
conv4_block4_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block4_2_bn[0][0]']
ation)
conv4_block4_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block4_2_relu[0][0]']
D)
conv4_block4_out (Add) (None, 14, 14, 1024) 0 ['conv4_block3_out[0][0]',
'conv4_block4_3_conv[0][0]']
conv4_block5_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block4_out[0][0]']
tchNormalization)
conv4_block5_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block5_preact_bn[0][0]
Activation) ']
conv4_block5_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block5_preact_relu[0][
D) 0]']
conv4_block5_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block5_1_conv[0][0]']
rmalization)
conv4_block5_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block5_1_bn[0][0]']
ation)
conv4_block5_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block5_1_relu[0][0]']
dding2D)
conv4_block5_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block5_2_pad[0][0]']
D)
conv4_block5_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block5_2_conv[0][0]']
rmalization)
conv4_block5_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block5_2_bn[0][0]']
ation)
conv4_block5_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block5_2_relu[0][0]']
D)
conv4_block5_out (Add) (None, 14, 14, 1024) 0 ['conv4_block4_out[0][0]',
'conv4_block5_3_conv[0][0]']
conv4_block6_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block5_out[0][0]']
tchNormalization)
conv4_block6_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block6_preact_bn[0][0]
Activation) ']
conv4_block6_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block6_preact_relu[0][
D) 0]']
conv4_block6_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block6_1_conv[0][0]']
rmalization)
conv4_block6_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block6_1_bn[0][0]']
ation)
conv4_block6_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block6_1_relu[0][0]']
dding2D)
conv4_block6_2_conv (Conv2 (None, 7, 7, 256) 589824 ['conv4_block6_2_pad[0][0]']
D)
conv4_block6_2_bn (BatchNo (None, 7, 7, 256) 1024 ['conv4_block6_2_conv[0][0]']
rmalization)
conv4_block6_2_relu (Activ (None, 7, 7, 256) 0 ['conv4_block6_2_bn[0][0]']
ation)
max_pooling2d_2 (MaxPoolin (None, 7, 7, 1024) 0 ['conv4_block5_out[0][0]']
g2D)
conv4_block6_3_conv (Conv2 (None, 7, 7, 1024) 263168 ['conv4_block6_2_relu[0][0]']
D)
conv4_block6_out (Add) (None, 7, 7, 1024) 0 ['max_pooling2d_2[0][0]',
'conv4_block6_3_conv[0][0]']
conv5_block1_preact_bn (Ba (None, 7, 7, 1024) 4096 ['conv4_block6_out[0][0]']
tchNormalization)
conv5_block1_preact_relu ( (None, 7, 7, 1024) 0 ['conv5_block1_preact_bn[0][0]
Activation) ']
conv5_block1_1_conv (Conv2 (None, 7, 7, 512) 524288 ['conv5_block1_preact_relu[0][
D) 0]']
conv5_block1_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block1_1_conv[0][0]']
rmalization)
conv5_block1_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block1_1_bn[0][0]']
ation)
conv5_block1_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block1_1_relu[0][0]']
dding2D)
conv5_block1_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block1_2_pad[0][0]']
D)
conv5_block1_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block1_2_conv[0][0]']
rmalization)
conv5_block1_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block1_2_bn[0][0]']
ation)
conv5_block1_0_conv (Conv2 (None, 7, 7, 2048) 2099200 ['conv5_block1_preact_relu[0][
D) 0]']
conv5_block1_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block1_2_relu[0][0]']
D)
conv5_block1_out (Add) (None, 7, 7, 2048) 0 ['conv5_block1_0_conv[0][0]',
'conv5_block1_3_conv[0][0]']
conv5_block2_preact_bn (Ba (None, 7, 7, 2048) 8192 ['conv5_block1_out[0][0]']
tchNormalization)
conv5_block2_preact_relu ( (None, 7, 7, 2048) 0 ['conv5_block2_preact_bn[0][0]
Activation) ']
conv5_block2_1_conv (Conv2 (None, 7, 7, 512) 1048576 ['conv5_block2_preact_relu[0][
D) 0]']
conv5_block2_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block2_1_conv[0][0]']
rmalization)
conv5_block2_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block2_1_bn[0][0]']
ation)
conv5_block2_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block2_1_relu[0][0]']
dding2D)
conv5_block2_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block2_2_pad[0][0]']
D)
conv5_block2_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block2_2_conv[0][0]']
rmalization)
conv5_block2_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block2_2_bn[0][0]']
ation)
conv5_block2_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block2_2_relu[0][0]']
D)
conv5_block2_out (Add) (None, 7, 7, 2048) 0 ['conv5_block1_out[0][0]',
'conv5_block2_3_conv[0][0]']
conv5_block3_preact_bn (Ba (None, 7, 7, 2048) 8192 ['conv5_block2_out[0][0]']
tchNormalization)
conv5_block3_preact_relu ( (None, 7, 7, 2048) 0 ['conv5_block3_preact_bn[0][0]
Activation) ']
conv5_block3_1_conv (Conv2 (None, 7, 7, 512) 1048576 ['conv5_block3_preact_relu[0][
D) 0]']
conv5_block3_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block3_1_conv[0][0]']
rmalization)
conv5_block3_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block3_1_bn[0][0]']
ation)
conv5_block3_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block3_1_relu[0][0]']
dding2D)
conv5_block3_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block3_2_pad[0][0]']
D)
conv5_block3_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block3_2_conv[0][0]']
rmalization)
conv5_block3_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block3_2_bn[0][0]']
ation)
conv5_block3_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block3_2_relu[0][0]']
D)
conv5_block3_out (Add) (None, 7, 7, 2048) 0 ['conv5_block2_out[0][0]',
'conv5_block3_3_conv[0][0]']
post_bn (BatchNormalizatio (None, 7, 7, 2048) 8192 ['conv5_block3_out[0][0]']
n)
post_relu (Activation) (None, 7, 7, 2048) 0 ['post_bn[0][0]']
avg_pool (GlobalAveragePoo (None, 2048) 0 ['post_relu[0][0]']
ling2D)
==================================================================================================
Total params: 23564800 (89.89 MB)
Trainable params: 23519360 (89.72 MB)
Non-trainable params: 45440 (177.50 KB)
__________________________________________________________________________________________________
# # Making all the layers of the Resnet model non-trainable. i.e. freezing them
for layer in resnet_model.layers:
layer.trainable = False
print(layer.name, layer.trainable)
input_1 False conv1_pad False conv1_conv False pool1_pad False pool1_pool False conv2_block1_preact_bn False conv2_block1_preact_relu False conv2_block1_1_conv False conv2_block1_1_bn False conv2_block1_1_relu False conv2_block1_2_pad False conv2_block1_2_conv False conv2_block1_2_bn False conv2_block1_2_relu False conv2_block1_0_conv False conv2_block1_3_conv False conv2_block1_out False conv2_block2_preact_bn False conv2_block2_preact_relu False conv2_block2_1_conv False conv2_block2_1_bn False conv2_block2_1_relu False conv2_block2_2_pad False conv2_block2_2_conv False conv2_block2_2_bn False conv2_block2_2_relu False conv2_block2_3_conv False conv2_block2_out False conv2_block3_preact_bn False conv2_block3_preact_relu False conv2_block3_1_conv False conv2_block3_1_bn False conv2_block3_1_relu False conv2_block3_2_pad False conv2_block3_2_conv False conv2_block3_2_bn False conv2_block3_2_relu False max_pooling2d False conv2_block3_3_conv False conv2_block3_out False conv3_block1_preact_bn False conv3_block1_preact_relu False conv3_block1_1_conv False conv3_block1_1_bn False conv3_block1_1_relu False conv3_block1_2_pad False conv3_block1_2_conv False conv3_block1_2_bn False conv3_block1_2_relu False conv3_block1_0_conv False conv3_block1_3_conv False conv3_block1_out False conv3_block2_preact_bn False conv3_block2_preact_relu False conv3_block2_1_conv False conv3_block2_1_bn False conv3_block2_1_relu False conv3_block2_2_pad False conv3_block2_2_conv False conv3_block2_2_bn False conv3_block2_2_relu False conv3_block2_3_conv False conv3_block2_out False conv3_block3_preact_bn False conv3_block3_preact_relu False conv3_block3_1_conv False conv3_block3_1_bn False conv3_block3_1_relu False conv3_block3_2_pad False conv3_block3_2_conv False conv3_block3_2_bn False conv3_block3_2_relu False conv3_block3_3_conv False conv3_block3_out False conv3_block4_preact_bn False conv3_block4_preact_relu False conv3_block4_1_conv False conv3_block4_1_bn False conv3_block4_1_relu False conv3_block4_2_pad False conv3_block4_2_conv False conv3_block4_2_bn False conv3_block4_2_relu False max_pooling2d_1 False conv3_block4_3_conv False conv3_block4_out False conv4_block1_preact_bn False conv4_block1_preact_relu False conv4_block1_1_conv False conv4_block1_1_bn False conv4_block1_1_relu False conv4_block1_2_pad False conv4_block1_2_conv False conv4_block1_2_bn False conv4_block1_2_relu False conv4_block1_0_conv False conv4_block1_3_conv False conv4_block1_out False conv4_block2_preact_bn False conv4_block2_preact_relu False conv4_block2_1_conv False conv4_block2_1_bn False conv4_block2_1_relu False conv4_block2_2_pad False conv4_block2_2_conv False conv4_block2_2_bn False conv4_block2_2_relu False conv4_block2_3_conv False conv4_block2_out False conv4_block3_preact_bn False conv4_block3_preact_relu False conv4_block3_1_conv False conv4_block3_1_bn False conv4_block3_1_relu False conv4_block3_2_pad False conv4_block3_2_conv False conv4_block3_2_bn False conv4_block3_2_relu False conv4_block3_3_conv False conv4_block3_out False conv4_block4_preact_bn False conv4_block4_preact_relu False conv4_block4_1_conv False conv4_block4_1_bn False conv4_block4_1_relu False conv4_block4_2_pad False conv4_block4_2_conv False conv4_block4_2_bn False conv4_block4_2_relu False conv4_block4_3_conv False conv4_block4_out False conv4_block5_preact_bn False conv4_block5_preact_relu False conv4_block5_1_conv False conv4_block5_1_bn False conv4_block5_1_relu False conv4_block5_2_pad False conv4_block5_2_conv False conv4_block5_2_bn False conv4_block5_2_relu False conv4_block5_3_conv False conv4_block5_out False conv4_block6_preact_bn False conv4_block6_preact_relu False conv4_block6_1_conv False conv4_block6_1_bn False conv4_block6_1_relu False conv4_block6_2_pad False conv4_block6_2_conv False conv4_block6_2_bn False conv4_block6_2_relu False max_pooling2d_2 False conv4_block6_3_conv False conv4_block6_out False conv5_block1_preact_bn False conv5_block1_preact_relu False conv5_block1_1_conv False conv5_block1_1_bn False conv5_block1_1_relu False conv5_block1_2_pad False conv5_block1_2_conv False conv5_block1_2_bn False conv5_block1_2_relu False conv5_block1_0_conv False conv5_block1_3_conv False conv5_block1_out False conv5_block2_preact_bn False conv5_block2_preact_relu False conv5_block2_1_conv False conv5_block2_1_bn False conv5_block2_1_relu False conv5_block2_2_pad False conv5_block2_2_conv False conv5_block2_2_bn False conv5_block2_2_relu False conv5_block2_3_conv False conv5_block2_out False conv5_block3_preact_bn False conv5_block3_preact_relu False conv5_block3_1_conv False conv5_block3_1_bn False conv5_block3_1_relu False conv5_block3_2_pad False conv5_block3_2_conv False conv5_block3_2_bn False conv5_block3_2_relu False conv5_block3_3_conv False conv5_block3_out False post_bn False post_relu False avg_pool False
# building CNN model with Resnet engine
# I will use same CNN structure and only replace the engine to Resnet
resnet_engine_model = Sequential()
# Adding the convolutional part of the Resnet created above
resnet_engine_model.add(resnet_model)
# Flattening the output of the Resnet engine
resnet_engine_model.add(Flatten())
# Adding a dense output layer
resnet_engine_model.add(Dense(32))
resnet_engine_model.add(LeakyReLU(0.1))
resnet_engine_model.add(Dropout(0.1))
resnet_engine_model.add(Dense(32))
resnet_engine_model.add(LeakyReLU(0.1))
resnet_engine_model.add(Dense(16))
resnet_engine_model.add(LeakyReLU(0.1))
resnet_engine_model.add(Dense(4, activation='softmax'))
# print model summary
resnet_engine_model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
model (Functional) (None, 2048) 23564800
flatten (Flatten) (None, 2048) 0
dense (Dense) (None, 32) 65568
leaky_re_lu (LeakyReLU) (None, 32) 0
dropout (Dropout) (None, 32) 0
dense_1 (Dense) (None, 32) 1056
leaky_re_lu_1 (LeakyReLU) (None, 32) 0
dense_2 (Dense) (None, 16) 528
leaky_re_lu_2 (LeakyReLU) (None, 16) 0
dense_3 (Dense) (None, 4) 68
=================================================================
Total params: 23632020 (90.15 MB)
Trainable params: 67220 (262.58 KB)
Non-trainable params: 23564800 (89.89 MB)
_________________________________________________________________
# compiling the model with Adam optimizer, categorical crossentropy loss function as this is multi class model
# and accuracy metrics to evaluate the models performance during traiing and validation
opt = Adam(learning_rate = 0.001) #use Adam optimizer with default learning rate
resnet_engine_model.compile(loss="categorical_crossentropy", optimizer=opt, metrics = ['accuracy'])
# Pulling a single large batch of random validatio data for validating after each epoch
valX, valY = validation_generator_rgb_2.next()
# the following code will stop the training if the val_loss continues above min val_loss for the run for more than 6 epochs
es = EarlyStopping(monitor = 'val_loss', mode = 'min', verbose = 1, patience = 6)
# Reduce LR on Plateau callback will reduce the learning rate to 0.2 of its current value, if val_loss does not improve for 5 consecutive epochs
learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_loss',
patience = 5,
verbose = 1,
factor = 0.2,
cooldown = 2,
min_lr = 0.0001)
resnet_hist = resnet_engine_model.fit(train_generator_rgb_2,
validation_data=(valX, valY),
epochs=20,
callbacks = [es, learning_rate_reduction])
Epoch 1/20 237/237 [==============================] - 51s 184ms/step - loss: 1.0260 - accuracy: 0.5609 - val_loss: 0.8536 - val_accuracy: 0.6016 - lr: 0.0010 Epoch 2/20 237/237 [==============================] - 39s 165ms/step - loss: 0.8386 - accuracy: 0.6588 - val_loss: 0.8266 - val_accuracy: 0.6641 - lr: 0.0010 Epoch 3/20 237/237 [==============================] - 38s 158ms/step - loss: 0.7801 - accuracy: 0.6834 - val_loss: 0.8735 - val_accuracy: 0.6484 - lr: 0.0010 Epoch 4/20 237/237 [==============================] - 36s 151ms/step - loss: 0.7307 - accuracy: 0.7067 - val_loss: 0.7787 - val_accuracy: 0.6797 - lr: 0.0010 Epoch 5/20 237/237 [==============================] - 35s 148ms/step - loss: 0.6919 - accuracy: 0.7227 - val_loss: 0.7597 - val_accuracy: 0.6875 - lr: 0.0010 Epoch 6/20 237/237 [==============================] - 32s 137ms/step - loss: 0.6651 - accuracy: 0.7344 - val_loss: 0.8204 - val_accuracy: 0.6719 - lr: 0.0010 Epoch 7/20 237/237 [==============================] - 29s 123ms/step - loss: 0.6242 - accuracy: 0.7507 - val_loss: 0.8408 - val_accuracy: 0.6719 - lr: 0.0010 Epoch 8/20 237/237 [==============================] - 33s 140ms/step - loss: 0.5970 - accuracy: 0.7661 - val_loss: 0.7630 - val_accuracy: 0.7109 - lr: 0.0010 Epoch 9/20 237/237 [==============================] - 32s 135ms/step - loss: 0.5664 - accuracy: 0.7768 - val_loss: 0.7813 - val_accuracy: 0.7109 - lr: 0.0010 Epoch 10/20 237/237 [==============================] - ETA: 0s - loss: 0.5420 - accuracy: 0.7881 Epoch 10: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 237/237 [==============================] - 28s 116ms/step - loss: 0.5420 - accuracy: 0.7881 - val_loss: 0.8142 - val_accuracy: 0.7109 - lr: 0.0010 Epoch 11/20 237/237 [==============================] - 32s 137ms/step - loss: 0.4623 - accuracy: 0.8254 - val_loss: 0.7464 - val_accuracy: 0.7188 - lr: 2.0000e-04 Epoch 12/20 237/237 [==============================] - 33s 139ms/step - loss: 0.4433 - accuracy: 0.8323 - val_loss: 0.7601 - val_accuracy: 0.7422 - lr: 2.0000e-04 Epoch 13/20 237/237 [==============================] - 29s 124ms/step - loss: 0.4293 - accuracy: 0.8353 - val_loss: 0.7424 - val_accuracy: 0.7344 - lr: 2.0000e-04 Epoch 14/20 237/237 [==============================] - 33s 137ms/step - loss: 0.4232 - accuracy: 0.8410 - val_loss: 0.7464 - val_accuracy: 0.7266 - lr: 2.0000e-04 Epoch 15/20 237/237 [==============================] - 26s 111ms/step - loss: 0.4071 - accuracy: 0.8466 - val_loss: 0.7656 - val_accuracy: 0.7266 - lr: 2.0000e-04 Epoch 16/20 237/237 [==============================] - 30s 125ms/step - loss: 0.4043 - accuracy: 0.8490 - val_loss: 0.7607 - val_accuracy: 0.7188 - lr: 2.0000e-04 Epoch 17/20 237/237 [==============================] - 33s 138ms/step - loss: 0.3927 - accuracy: 0.8538 - val_loss: 0.7551 - val_accuracy: 0.7109 - lr: 2.0000e-04 Epoch 18/20 237/237 [==============================] - ETA: 0s - loss: 0.3863 - accuracy: 0.8559 Epoch 18: ReduceLROnPlateau reducing learning rate to 0.0001. 237/237 [==============================] - 26s 111ms/step - loss: 0.3863 - accuracy: 0.8559 - val_loss: 0.7656 - val_accuracy: 0.7344 - lr: 2.0000e-04 Epoch 19/20 237/237 [==============================] - 32s 133ms/step - loss: 0.3637 - accuracy: 0.8681 - val_loss: 0.7441 - val_accuracy: 0.7266 - lr: 1.0000e-04 Epoch 19: early stopping
# evaluate vgg_engine_model on test data
resnet_engine_model.evaluate(test_generator_rgb_2)
1/1 [==============================] - 0s 360ms/step - loss: 0.8136 - accuracy: 0.7266
[0.8135553598403931, 0.7265625]
plot_accuracy(resnet_hist)
plot_loss(resnet_hist)
# get test images and their classes
test_images, test_classes = next(test_generator_rgb_2)
actual_classes = np.argmax(test_classes, axis = -1)
# Make predictions
resnet_predictions = resnet_engine_model.predict(test_images)
# Convert predictions to class labels
resnet_predictions = np.argmax(resnet_predictions, axis = -1)
# generate classification report and confussion matrix
metrics_score(actual_classes,resnet_predictions )
4/4 [==============================] - 1s 31ms/step
precision recall f1-score support
0 0.84 0.81 0.83 32
1 0.50 0.62 0.56 32
2 0.71 0.62 0.67 32
3 0.93 0.84 0.89 32
accuracy 0.73 128
macro avg 0.75 0.73 0.73 128
weighted avg 0.75 0.73 0.73 128
Observations and Insights:
Note: You can even go back and build your own architecture on top of the ResNet Transfer layer and see if you can improve the performance.
# clear the previous model's history from the Keras backend
from tensorflow.keras import backend
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(666)
import random
random.seed(666)
tf.random.set_seed(666)
# instantiate EffectiveNetB0 model
model = EfficientNetB0(weights='imagenet')
model.summary()
Model: "efficientnet-b0"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0 []
stem_conv (Conv2D) (None, 112, 112, 32) 864 ['input_1[0][0]']
stem_bn (BatchNormalizatio (None, 112, 112, 32) 128 ['stem_conv[0][0]']
n)
stem_activation (Activatio (None, 112, 112, 32) 0 ['stem_bn[0][0]']
n)
block1a_dwconv (DepthwiseC (None, 112, 112, 32) 288 ['stem_activation[0][0]']
onv2D)
block1a_bn (BatchNormaliza (None, 112, 112, 32) 128 ['block1a_dwconv[0][0]']
tion)
block1a_activation (Activa (None, 112, 112, 32) 0 ['block1a_bn[0][0]']
tion)
block1a_se_squeeze (Global (None, 32) 0 ['block1a_activation[0][0]']
AveragePooling2D)
block1a_se_reshape (Reshap (None, 1, 1, 32) 0 ['block1a_se_squeeze[0][0]']
e)
block1a_se_reduce (Conv2D) (None, 1, 1, 8) 264 ['block1a_se_reshape[0][0]']
block1a_se_expand (Conv2D) (None, 1, 1, 32) 288 ['block1a_se_reduce[0][0]']
block1a_se_excite (Multipl (None, 112, 112, 32) 0 ['block1a_activation[0][0]',
y) 'block1a_se_expand[0][0]']
block1a_project_conv (Conv (None, 112, 112, 16) 512 ['block1a_se_excite[0][0]']
2D)
block1a_project_bn (BatchN (None, 112, 112, 16) 64 ['block1a_project_conv[0][0]']
ormalization)
block2a_expand_conv (Conv2 (None, 112, 112, 96) 1536 ['block1a_project_bn[0][0]']
D)
block2a_expand_bn (BatchNo (None, 112, 112, 96) 384 ['block2a_expand_conv[0][0]']
rmalization)
block2a_expand_activation (None, 112, 112, 96) 0 ['block2a_expand_bn[0][0]']
(Activation)
block2a_dwconv (DepthwiseC (None, 56, 56, 96) 864 ['block2a_expand_activation[0]
onv2D) [0]']
block2a_bn (BatchNormaliza (None, 56, 56, 96) 384 ['block2a_dwconv[0][0]']
tion)
block2a_activation (Activa (None, 56, 56, 96) 0 ['block2a_bn[0][0]']
tion)
block2a_se_squeeze (Global (None, 96) 0 ['block2a_activation[0][0]']
AveragePooling2D)
block2a_se_reshape (Reshap (None, 1, 1, 96) 0 ['block2a_se_squeeze[0][0]']
e)
block2a_se_reduce (Conv2D) (None, 1, 1, 4) 388 ['block2a_se_reshape[0][0]']
block2a_se_expand (Conv2D) (None, 1, 1, 96) 480 ['block2a_se_reduce[0][0]']
block2a_se_excite (Multipl (None, 56, 56, 96) 0 ['block2a_activation[0][0]',
y) 'block2a_se_expand[0][0]']
block2a_project_conv (Conv (None, 56, 56, 24) 2304 ['block2a_se_excite[0][0]']
2D)
block2a_project_bn (BatchN (None, 56, 56, 24) 96 ['block2a_project_conv[0][0]']
ormalization)
block2b_expand_conv (Conv2 (None, 56, 56, 144) 3456 ['block2a_project_bn[0][0]']
D)
block2b_expand_bn (BatchNo (None, 56, 56, 144) 576 ['block2b_expand_conv[0][0]']
rmalization)
block2b_expand_activation (None, 56, 56, 144) 0 ['block2b_expand_bn[0][0]']
(Activation)
block2b_dwconv (DepthwiseC (None, 56, 56, 144) 1296 ['block2b_expand_activation[0]
onv2D) [0]']
block2b_bn (BatchNormaliza (None, 56, 56, 144) 576 ['block2b_dwconv[0][0]']
tion)
block2b_activation (Activa (None, 56, 56, 144) 0 ['block2b_bn[0][0]']
tion)
block2b_se_squeeze (Global (None, 144) 0 ['block2b_activation[0][0]']
AveragePooling2D)
block2b_se_reshape (Reshap (None, 1, 1, 144) 0 ['block2b_se_squeeze[0][0]']
e)
block2b_se_reduce (Conv2D) (None, 1, 1, 6) 870 ['block2b_se_reshape[0][0]']
block2b_se_expand (Conv2D) (None, 1, 1, 144) 1008 ['block2b_se_reduce[0][0]']
block2b_se_excite (Multipl (None, 56, 56, 144) 0 ['block2b_activation[0][0]',
y) 'block2b_se_expand[0][0]']
block2b_project_conv (Conv (None, 56, 56, 24) 3456 ['block2b_se_excite[0][0]']
2D)
block2b_project_bn (BatchN (None, 56, 56, 24) 96 ['block2b_project_conv[0][0]']
ormalization)
block2b_drop (FixedDropout (None, 56, 56, 24) 0 ['block2b_project_bn[0][0]']
)
block2b_add (Add) (None, 56, 56, 24) 0 ['block2b_drop[0][0]',
'block2a_project_bn[0][0]']
block3a_expand_conv (Conv2 (None, 56, 56, 144) 3456 ['block2b_add[0][0]']
D)
block3a_expand_bn (BatchNo (None, 56, 56, 144) 576 ['block3a_expand_conv[0][0]']
rmalization)
block3a_expand_activation (None, 56, 56, 144) 0 ['block3a_expand_bn[0][0]']
(Activation)
block3a_dwconv (DepthwiseC (None, 28, 28, 144) 3600 ['block3a_expand_activation[0]
onv2D) [0]']
block3a_bn (BatchNormaliza (None, 28, 28, 144) 576 ['block3a_dwconv[0][0]']
tion)
block3a_activation (Activa (None, 28, 28, 144) 0 ['block3a_bn[0][0]']
tion)
block3a_se_squeeze (Global (None, 144) 0 ['block3a_activation[0][0]']
AveragePooling2D)
block3a_se_reshape (Reshap (None, 1, 1, 144) 0 ['block3a_se_squeeze[0][0]']
e)
block3a_se_reduce (Conv2D) (None, 1, 1, 6) 870 ['block3a_se_reshape[0][0]']
block3a_se_expand (Conv2D) (None, 1, 1, 144) 1008 ['block3a_se_reduce[0][0]']
block3a_se_excite (Multipl (None, 28, 28, 144) 0 ['block3a_activation[0][0]',
y) 'block3a_se_expand[0][0]']
block3a_project_conv (Conv (None, 28, 28, 40) 5760 ['block3a_se_excite[0][0]']
2D)
block3a_project_bn (BatchN (None, 28, 28, 40) 160 ['block3a_project_conv[0][0]']
ormalization)
block3b_expand_conv (Conv2 (None, 28, 28, 240) 9600 ['block3a_project_bn[0][0]']
D)
block3b_expand_bn (BatchNo (None, 28, 28, 240) 960 ['block3b_expand_conv[0][0]']
rmalization)
block3b_expand_activation (None, 28, 28, 240) 0 ['block3b_expand_bn[0][0]']
(Activation)
block3b_dwconv (DepthwiseC (None, 28, 28, 240) 6000 ['block3b_expand_activation[0]
onv2D) [0]']
block3b_bn (BatchNormaliza (None, 28, 28, 240) 960 ['block3b_dwconv[0][0]']
tion)
block3b_activation (Activa (None, 28, 28, 240) 0 ['block3b_bn[0][0]']
tion)
block3b_se_squeeze (Global (None, 240) 0 ['block3b_activation[0][0]']
AveragePooling2D)
block3b_se_reshape (Reshap (None, 1, 1, 240) 0 ['block3b_se_squeeze[0][0]']
e)
block3b_se_reduce (Conv2D) (None, 1, 1, 10) 2410 ['block3b_se_reshape[0][0]']
block3b_se_expand (Conv2D) (None, 1, 1, 240) 2640 ['block3b_se_reduce[0][0]']
block3b_se_excite (Multipl (None, 28, 28, 240) 0 ['block3b_activation[0][0]',
y) 'block3b_se_expand[0][0]']
block3b_project_conv (Conv (None, 28, 28, 40) 9600 ['block3b_se_excite[0][0]']
2D)
block3b_project_bn (BatchN (None, 28, 28, 40) 160 ['block3b_project_conv[0][0]']
ormalization)
block3b_drop (FixedDropout (None, 28, 28, 40) 0 ['block3b_project_bn[0][0]']
)
block3b_add (Add) (None, 28, 28, 40) 0 ['block3b_drop[0][0]',
'block3a_project_bn[0][0]']
block4a_expand_conv (Conv2 (None, 28, 28, 240) 9600 ['block3b_add[0][0]']
D)
block4a_expand_bn (BatchNo (None, 28, 28, 240) 960 ['block4a_expand_conv[0][0]']
rmalization)
block4a_expand_activation (None, 28, 28, 240) 0 ['block4a_expand_bn[0][0]']
(Activation)
block4a_dwconv (DepthwiseC (None, 14, 14, 240) 2160 ['block4a_expand_activation[0]
onv2D) [0]']
block4a_bn (BatchNormaliza (None, 14, 14, 240) 960 ['block4a_dwconv[0][0]']
tion)
block4a_activation (Activa (None, 14, 14, 240) 0 ['block4a_bn[0][0]']
tion)
block4a_se_squeeze (Global (None, 240) 0 ['block4a_activation[0][0]']
AveragePooling2D)
block4a_se_reshape (Reshap (None, 1, 1, 240) 0 ['block4a_se_squeeze[0][0]']
e)
block4a_se_reduce (Conv2D) (None, 1, 1, 10) 2410 ['block4a_se_reshape[0][0]']
block4a_se_expand (Conv2D) (None, 1, 1, 240) 2640 ['block4a_se_reduce[0][0]']
block4a_se_excite (Multipl (None, 14, 14, 240) 0 ['block4a_activation[0][0]',
y) 'block4a_se_expand[0][0]']
block4a_project_conv (Conv (None, 14, 14, 80) 19200 ['block4a_se_excite[0][0]']
2D)
block4a_project_bn (BatchN (None, 14, 14, 80) 320 ['block4a_project_conv[0][0]']
ormalization)
block4b_expand_conv (Conv2 (None, 14, 14, 480) 38400 ['block4a_project_bn[0][0]']
D)
block4b_expand_bn (BatchNo (None, 14, 14, 480) 1920 ['block4b_expand_conv[0][0]']
rmalization)
block4b_expand_activation (None, 14, 14, 480) 0 ['block4b_expand_bn[0][0]']
(Activation)
block4b_dwconv (DepthwiseC (None, 14, 14, 480) 4320 ['block4b_expand_activation[0]
onv2D) [0]']
block4b_bn (BatchNormaliza (None, 14, 14, 480) 1920 ['block4b_dwconv[0][0]']
tion)
block4b_activation (Activa (None, 14, 14, 480) 0 ['block4b_bn[0][0]']
tion)
block4b_se_squeeze (Global (None, 480) 0 ['block4b_activation[0][0]']
AveragePooling2D)
block4b_se_reshape (Reshap (None, 1, 1, 480) 0 ['block4b_se_squeeze[0][0]']
e)
block4b_se_reduce (Conv2D) (None, 1, 1, 20) 9620 ['block4b_se_reshape[0][0]']
block4b_se_expand (Conv2D) (None, 1, 1, 480) 10080 ['block4b_se_reduce[0][0]']
block4b_se_excite (Multipl (None, 14, 14, 480) 0 ['block4b_activation[0][0]',
y) 'block4b_se_expand[0][0]']
block4b_project_conv (Conv (None, 14, 14, 80) 38400 ['block4b_se_excite[0][0]']
2D)
block4b_project_bn (BatchN (None, 14, 14, 80) 320 ['block4b_project_conv[0][0]']
ormalization)
block4b_drop (FixedDropout (None, 14, 14, 80) 0 ['block4b_project_bn[0][0]']
)
block4b_add (Add) (None, 14, 14, 80) 0 ['block4b_drop[0][0]',
'block4a_project_bn[0][0]']
block4c_expand_conv (Conv2 (None, 14, 14, 480) 38400 ['block4b_add[0][0]']
D)
block4c_expand_bn (BatchNo (None, 14, 14, 480) 1920 ['block4c_expand_conv[0][0]']
rmalization)
block4c_expand_activation (None, 14, 14, 480) 0 ['block4c_expand_bn[0][0]']
(Activation)
block4c_dwconv (DepthwiseC (None, 14, 14, 480) 4320 ['block4c_expand_activation[0]
onv2D) [0]']
block4c_bn (BatchNormaliza (None, 14, 14, 480) 1920 ['block4c_dwconv[0][0]']
tion)
block4c_activation (Activa (None, 14, 14, 480) 0 ['block4c_bn[0][0]']
tion)
block4c_se_squeeze (Global (None, 480) 0 ['block4c_activation[0][0]']
AveragePooling2D)
block4c_se_reshape (Reshap (None, 1, 1, 480) 0 ['block4c_se_squeeze[0][0]']
e)
block4c_se_reduce (Conv2D) (None, 1, 1, 20) 9620 ['block4c_se_reshape[0][0]']
block4c_se_expand (Conv2D) (None, 1, 1, 480) 10080 ['block4c_se_reduce[0][0]']
block4c_se_excite (Multipl (None, 14, 14, 480) 0 ['block4c_activation[0][0]',
y) 'block4c_se_expand[0][0]']
block4c_project_conv (Conv (None, 14, 14, 80) 38400 ['block4c_se_excite[0][0]']
2D)
block4c_project_bn (BatchN (None, 14, 14, 80) 320 ['block4c_project_conv[0][0]']
ormalization)
block4c_drop (FixedDropout (None, 14, 14, 80) 0 ['block4c_project_bn[0][0]']
)
block4c_add (Add) (None, 14, 14, 80) 0 ['block4c_drop[0][0]',
'block4b_add[0][0]']
block5a_expand_conv (Conv2 (None, 14, 14, 480) 38400 ['block4c_add[0][0]']
D)
block5a_expand_bn (BatchNo (None, 14, 14, 480) 1920 ['block5a_expand_conv[0][0]']
rmalization)
block5a_expand_activation (None, 14, 14, 480) 0 ['block5a_expand_bn[0][0]']
(Activation)
block5a_dwconv (DepthwiseC (None, 14, 14, 480) 12000 ['block5a_expand_activation[0]
onv2D) [0]']
block5a_bn (BatchNormaliza (None, 14, 14, 480) 1920 ['block5a_dwconv[0][0]']
tion)
block5a_activation (Activa (None, 14, 14, 480) 0 ['block5a_bn[0][0]']
tion)
block5a_se_squeeze (Global (None, 480) 0 ['block5a_activation[0][0]']
AveragePooling2D)
block5a_se_reshape (Reshap (None, 1, 1, 480) 0 ['block5a_se_squeeze[0][0]']
e)
block5a_se_reduce (Conv2D) (None, 1, 1, 20) 9620 ['block5a_se_reshape[0][0]']
block5a_se_expand (Conv2D) (None, 1, 1, 480) 10080 ['block5a_se_reduce[0][0]']
block5a_se_excite (Multipl (None, 14, 14, 480) 0 ['block5a_activation[0][0]',
y) 'block5a_se_expand[0][0]']
block5a_project_conv (Conv (None, 14, 14, 112) 53760 ['block5a_se_excite[0][0]']
2D)
block5a_project_bn (BatchN (None, 14, 14, 112) 448 ['block5a_project_conv[0][0]']
ormalization)
block5b_expand_conv (Conv2 (None, 14, 14, 672) 75264 ['block5a_project_bn[0][0]']
D)
block5b_expand_bn (BatchNo (None, 14, 14, 672) 2688 ['block5b_expand_conv[0][0]']
rmalization)
block5b_expand_activation (None, 14, 14, 672) 0 ['block5b_expand_bn[0][0]']
(Activation)
block5b_dwconv (DepthwiseC (None, 14, 14, 672) 16800 ['block5b_expand_activation[0]
onv2D) [0]']
block5b_bn (BatchNormaliza (None, 14, 14, 672) 2688 ['block5b_dwconv[0][0]']
tion)
block5b_activation (Activa (None, 14, 14, 672) 0 ['block5b_bn[0][0]']
tion)
block5b_se_squeeze (Global (None, 672) 0 ['block5b_activation[0][0]']
AveragePooling2D)
block5b_se_reshape (Reshap (None, 1, 1, 672) 0 ['block5b_se_squeeze[0][0]']
e)
block5b_se_reduce (Conv2D) (None, 1, 1, 28) 18844 ['block5b_se_reshape[0][0]']
block5b_se_expand (Conv2D) (None, 1, 1, 672) 19488 ['block5b_se_reduce[0][0]']
block5b_se_excite (Multipl (None, 14, 14, 672) 0 ['block5b_activation[0][0]',
y) 'block5b_se_expand[0][0]']
block5b_project_conv (Conv (None, 14, 14, 112) 75264 ['block5b_se_excite[0][0]']
2D)
block5b_project_bn (BatchN (None, 14, 14, 112) 448 ['block5b_project_conv[0][0]']
ormalization)
block5b_drop (FixedDropout (None, 14, 14, 112) 0 ['block5b_project_bn[0][0]']
)
block5b_add (Add) (None, 14, 14, 112) 0 ['block5b_drop[0][0]',
'block5a_project_bn[0][0]']
block5c_expand_conv (Conv2 (None, 14, 14, 672) 75264 ['block5b_add[0][0]']
D)
block5c_expand_bn (BatchNo (None, 14, 14, 672) 2688 ['block5c_expand_conv[0][0]']
rmalization)
block5c_expand_activation (None, 14, 14, 672) 0 ['block5c_expand_bn[0][0]']
(Activation)
block5c_dwconv (DepthwiseC (None, 14, 14, 672) 16800 ['block5c_expand_activation[0]
onv2D) [0]']
block5c_bn (BatchNormaliza (None, 14, 14, 672) 2688 ['block5c_dwconv[0][0]']
tion)
block5c_activation (Activa (None, 14, 14, 672) 0 ['block5c_bn[0][0]']
tion)
block5c_se_squeeze (Global (None, 672) 0 ['block5c_activation[0][0]']
AveragePooling2D)
block5c_se_reshape (Reshap (None, 1, 1, 672) 0 ['block5c_se_squeeze[0][0]']
e)
block5c_se_reduce (Conv2D) (None, 1, 1, 28) 18844 ['block5c_se_reshape[0][0]']
block5c_se_expand (Conv2D) (None, 1, 1, 672) 19488 ['block5c_se_reduce[0][0]']
block5c_se_excite (Multipl (None, 14, 14, 672) 0 ['block5c_activation[0][0]',
y) 'block5c_se_expand[0][0]']
block5c_project_conv (Conv (None, 14, 14, 112) 75264 ['block5c_se_excite[0][0]']
2D)
block5c_project_bn (BatchN (None, 14, 14, 112) 448 ['block5c_project_conv[0][0]']
ormalization)
block5c_drop (FixedDropout (None, 14, 14, 112) 0 ['block5c_project_bn[0][0]']
)
block5c_add (Add) (None, 14, 14, 112) 0 ['block5c_drop[0][0]',
'block5b_add[0][0]']
block6a_expand_conv (Conv2 (None, 14, 14, 672) 75264 ['block5c_add[0][0]']
D)
block6a_expand_bn (BatchNo (None, 14, 14, 672) 2688 ['block6a_expand_conv[0][0]']
rmalization)
block6a_expand_activation (None, 14, 14, 672) 0 ['block6a_expand_bn[0][0]']
(Activation)
block6a_dwconv (DepthwiseC (None, 7, 7, 672) 16800 ['block6a_expand_activation[0]
onv2D) [0]']
block6a_bn (BatchNormaliza (None, 7, 7, 672) 2688 ['block6a_dwconv[0][0]']
tion)
block6a_activation (Activa (None, 7, 7, 672) 0 ['block6a_bn[0][0]']
tion)
block6a_se_squeeze (Global (None, 672) 0 ['block6a_activation[0][0]']
AveragePooling2D)
block6a_se_reshape (Reshap (None, 1, 1, 672) 0 ['block6a_se_squeeze[0][0]']
e)
block6a_se_reduce (Conv2D) (None, 1, 1, 28) 18844 ['block6a_se_reshape[0][0]']
block6a_se_expand (Conv2D) (None, 1, 1, 672) 19488 ['block6a_se_reduce[0][0]']
block6a_se_excite (Multipl (None, 7, 7, 672) 0 ['block6a_activation[0][0]',
y) 'block6a_se_expand[0][0]']
block6a_project_conv (Conv (None, 7, 7, 192) 129024 ['block6a_se_excite[0][0]']
2D)
block6a_project_bn (BatchN (None, 7, 7, 192) 768 ['block6a_project_conv[0][0]']
ormalization)
block6b_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6a_project_bn[0][0]']
D)
block6b_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block6b_expand_conv[0][0]']
rmalization)
block6b_expand_activation (None, 7, 7, 1152) 0 ['block6b_expand_bn[0][0]']
(Activation)
block6b_dwconv (DepthwiseC (None, 7, 7, 1152) 28800 ['block6b_expand_activation[0]
onv2D) [0]']
block6b_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block6b_dwconv[0][0]']
tion)
block6b_activation (Activa (None, 7, 7, 1152) 0 ['block6b_bn[0][0]']
tion)
block6b_se_squeeze (Global (None, 1152) 0 ['block6b_activation[0][0]']
AveragePooling2D)
block6b_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block6b_se_squeeze[0][0]']
e)
block6b_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block6b_se_reshape[0][0]']
block6b_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block6b_se_reduce[0][0]']
block6b_se_excite (Multipl (None, 7, 7, 1152) 0 ['block6b_activation[0][0]',
y) 'block6b_se_expand[0][0]']
block6b_project_conv (Conv (None, 7, 7, 192) 221184 ['block6b_se_excite[0][0]']
2D)
block6b_project_bn (BatchN (None, 7, 7, 192) 768 ['block6b_project_conv[0][0]']
ormalization)
block6b_drop (FixedDropout (None, 7, 7, 192) 0 ['block6b_project_bn[0][0]']
)
block6b_add (Add) (None, 7, 7, 192) 0 ['block6b_drop[0][0]',
'block6a_project_bn[0][0]']
block6c_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6b_add[0][0]']
D)
block6c_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block6c_expand_conv[0][0]']
rmalization)
block6c_expand_activation (None, 7, 7, 1152) 0 ['block6c_expand_bn[0][0]']
(Activation)
block6c_dwconv (DepthwiseC (None, 7, 7, 1152) 28800 ['block6c_expand_activation[0]
onv2D) [0]']
block6c_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block6c_dwconv[0][0]']
tion)
block6c_activation (Activa (None, 7, 7, 1152) 0 ['block6c_bn[0][0]']
tion)
block6c_se_squeeze (Global (None, 1152) 0 ['block6c_activation[0][0]']
AveragePooling2D)
block6c_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block6c_se_squeeze[0][0]']
e)
block6c_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block6c_se_reshape[0][0]']
block6c_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block6c_se_reduce[0][0]']
block6c_se_excite (Multipl (None, 7, 7, 1152) 0 ['block6c_activation[0][0]',
y) 'block6c_se_expand[0][0]']
block6c_project_conv (Conv (None, 7, 7, 192) 221184 ['block6c_se_excite[0][0]']
2D)
block6c_project_bn (BatchN (None, 7, 7, 192) 768 ['block6c_project_conv[0][0]']
ormalization)
block6c_drop (FixedDropout (None, 7, 7, 192) 0 ['block6c_project_bn[0][0]']
)
block6c_add (Add) (None, 7, 7, 192) 0 ['block6c_drop[0][0]',
'block6b_add[0][0]']
block6d_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6c_add[0][0]']
D)
block6d_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block6d_expand_conv[0][0]']
rmalization)
block6d_expand_activation (None, 7, 7, 1152) 0 ['block6d_expand_bn[0][0]']
(Activation)
block6d_dwconv (DepthwiseC (None, 7, 7, 1152) 28800 ['block6d_expand_activation[0]
onv2D) [0]']
block6d_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block6d_dwconv[0][0]']
tion)
block6d_activation (Activa (None, 7, 7, 1152) 0 ['block6d_bn[0][0]']
tion)
block6d_se_squeeze (Global (None, 1152) 0 ['block6d_activation[0][0]']
AveragePooling2D)
block6d_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block6d_se_squeeze[0][0]']
e)
block6d_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block6d_se_reshape[0][0]']
block6d_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block6d_se_reduce[0][0]']
block6d_se_excite (Multipl (None, 7, 7, 1152) 0 ['block6d_activation[0][0]',
y) 'block6d_se_expand[0][0]']
block6d_project_conv (Conv (None, 7, 7, 192) 221184 ['block6d_se_excite[0][0]']
2D)
block6d_project_bn (BatchN (None, 7, 7, 192) 768 ['block6d_project_conv[0][0]']
ormalization)
block6d_drop (FixedDropout (None, 7, 7, 192) 0 ['block6d_project_bn[0][0]']
)
block6d_add (Add) (None, 7, 7, 192) 0 ['block6d_drop[0][0]',
'block6c_add[0][0]']
block7a_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6d_add[0][0]']
D)
block7a_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block7a_expand_conv[0][0]']
rmalization)
block7a_expand_activation (None, 7, 7, 1152) 0 ['block7a_expand_bn[0][0]']
(Activation)
block7a_dwconv (DepthwiseC (None, 7, 7, 1152) 10368 ['block7a_expand_activation[0]
onv2D) [0]']
block7a_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block7a_dwconv[0][0]']
tion)
block7a_activation (Activa (None, 7, 7, 1152) 0 ['block7a_bn[0][0]']
tion)
block7a_se_squeeze (Global (None, 1152) 0 ['block7a_activation[0][0]']
AveragePooling2D)
block7a_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block7a_se_squeeze[0][0]']
e)
block7a_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block7a_se_reshape[0][0]']
block7a_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block7a_se_reduce[0][0]']
block7a_se_excite (Multipl (None, 7, 7, 1152) 0 ['block7a_activation[0][0]',
y) 'block7a_se_expand[0][0]']
block7a_project_conv (Conv (None, 7, 7, 320) 368640 ['block7a_se_excite[0][0]']
2D)
block7a_project_bn (BatchN (None, 7, 7, 320) 1280 ['block7a_project_conv[0][0]']
ormalization)
top_conv (Conv2D) (None, 7, 7, 1280) 409600 ['block7a_project_bn[0][0]']
top_bn (BatchNormalization (None, 7, 7, 1280) 5120 ['top_conv[0][0]']
)
top_activation (Activation (None, 7, 7, 1280) 0 ['top_bn[0][0]']
)
avg_pool (GlobalAveragePoo (None, 1280) 0 ['top_activation[0][0]']
ling2D)
top_dropout (Dropout) (None, 1280) 0 ['avg_pool[0][0]']
probs (Dense) (None, 1000) 1281000 ['top_dropout[0][0]']
==================================================================================================
Total params: 5330564 (20.33 MB)
Trainable params: 5288548 (20.17 MB)
Non-trainable params: 42016 (164.12 KB)
__________________________________________________________________________________________________
# again, I will use the entire EfficientNet architechture up to top_dropout layer
transfer_layer = model.get_layer('top_dropout')
effnet_model = Model(inputs=model.input, outputs=transfer_layer.output)
effnet_model.summary()
Model: "model"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0 []
stem_conv (Conv2D) (None, 112, 112, 32) 864 ['input_1[0][0]']
stem_bn (BatchNormalizatio (None, 112, 112, 32) 128 ['stem_conv[0][0]']
n)
stem_activation (Activatio (None, 112, 112, 32) 0 ['stem_bn[0][0]']
n)
block1a_dwconv (DepthwiseC (None, 112, 112, 32) 288 ['stem_activation[0][0]']
onv2D)
block1a_bn (BatchNormaliza (None, 112, 112, 32) 128 ['block1a_dwconv[0][0]']
tion)
block1a_activation (Activa (None, 112, 112, 32) 0 ['block1a_bn[0][0]']
tion)
block1a_se_squeeze (Global (None, 32) 0 ['block1a_activation[0][0]']
AveragePooling2D)
block1a_se_reshape (Reshap (None, 1, 1, 32) 0 ['block1a_se_squeeze[0][0]']
e)
block1a_se_reduce (Conv2D) (None, 1, 1, 8) 264 ['block1a_se_reshape[0][0]']
block1a_se_expand (Conv2D) (None, 1, 1, 32) 288 ['block1a_se_reduce[0][0]']
block1a_se_excite (Multipl (None, 112, 112, 32) 0 ['block1a_activation[0][0]',
y) 'block1a_se_expand[0][0]']
block1a_project_conv (Conv (None, 112, 112, 16) 512 ['block1a_se_excite[0][0]']
2D)
block1a_project_bn (BatchN (None, 112, 112, 16) 64 ['block1a_project_conv[0][0]']
ormalization)
block2a_expand_conv (Conv2 (None, 112, 112, 96) 1536 ['block1a_project_bn[0][0]']
D)
block2a_expand_bn (BatchNo (None, 112, 112, 96) 384 ['block2a_expand_conv[0][0]']
rmalization)
block2a_expand_activation (None, 112, 112, 96) 0 ['block2a_expand_bn[0][0]']
(Activation)
block2a_dwconv (DepthwiseC (None, 56, 56, 96) 864 ['block2a_expand_activation[0]
onv2D) [0]']
block2a_bn (BatchNormaliza (None, 56, 56, 96) 384 ['block2a_dwconv[0][0]']
tion)
block2a_activation (Activa (None, 56, 56, 96) 0 ['block2a_bn[0][0]']
tion)
block2a_se_squeeze (Global (None, 96) 0 ['block2a_activation[0][0]']
AveragePooling2D)
block2a_se_reshape (Reshap (None, 1, 1, 96) 0 ['block2a_se_squeeze[0][0]']
e)
block2a_se_reduce (Conv2D) (None, 1, 1, 4) 388 ['block2a_se_reshape[0][0]']
block2a_se_expand (Conv2D) (None, 1, 1, 96) 480 ['block2a_se_reduce[0][0]']
block2a_se_excite (Multipl (None, 56, 56, 96) 0 ['block2a_activation[0][0]',
y) 'block2a_se_expand[0][0]']
block2a_project_conv (Conv (None, 56, 56, 24) 2304 ['block2a_se_excite[0][0]']
2D)
block2a_project_bn (BatchN (None, 56, 56, 24) 96 ['block2a_project_conv[0][0]']
ormalization)
block2b_expand_conv (Conv2 (None, 56, 56, 144) 3456 ['block2a_project_bn[0][0]']
D)
block2b_expand_bn (BatchNo (None, 56, 56, 144) 576 ['block2b_expand_conv[0][0]']
rmalization)
block2b_expand_activation (None, 56, 56, 144) 0 ['block2b_expand_bn[0][0]']
(Activation)
block2b_dwconv (DepthwiseC (None, 56, 56, 144) 1296 ['block2b_expand_activation[0]
onv2D) [0]']
block2b_bn (BatchNormaliza (None, 56, 56, 144) 576 ['block2b_dwconv[0][0]']
tion)
block2b_activation (Activa (None, 56, 56, 144) 0 ['block2b_bn[0][0]']
tion)
block2b_se_squeeze (Global (None, 144) 0 ['block2b_activation[0][0]']
AveragePooling2D)
block2b_se_reshape (Reshap (None, 1, 1, 144) 0 ['block2b_se_squeeze[0][0]']
e)
block2b_se_reduce (Conv2D) (None, 1, 1, 6) 870 ['block2b_se_reshape[0][0]']
block2b_se_expand (Conv2D) (None, 1, 1, 144) 1008 ['block2b_se_reduce[0][0]']
block2b_se_excite (Multipl (None, 56, 56, 144) 0 ['block2b_activation[0][0]',
y) 'block2b_se_expand[0][0]']
block2b_project_conv (Conv (None, 56, 56, 24) 3456 ['block2b_se_excite[0][0]']
2D)
block2b_project_bn (BatchN (None, 56, 56, 24) 96 ['block2b_project_conv[0][0]']
ormalization)
block2b_drop (FixedDropout (None, 56, 56, 24) 0 ['block2b_project_bn[0][0]']
)
block2b_add (Add) (None, 56, 56, 24) 0 ['block2b_drop[0][0]',
'block2a_project_bn[0][0]']
block3a_expand_conv (Conv2 (None, 56, 56, 144) 3456 ['block2b_add[0][0]']
D)
block3a_expand_bn (BatchNo (None, 56, 56, 144) 576 ['block3a_expand_conv[0][0]']
rmalization)
block3a_expand_activation (None, 56, 56, 144) 0 ['block3a_expand_bn[0][0]']
(Activation)
block3a_dwconv (DepthwiseC (None, 28, 28, 144) 3600 ['block3a_expand_activation[0]
onv2D) [0]']
block3a_bn (BatchNormaliza (None, 28, 28, 144) 576 ['block3a_dwconv[0][0]']
tion)
block3a_activation (Activa (None, 28, 28, 144) 0 ['block3a_bn[0][0]']
tion)
block3a_se_squeeze (Global (None, 144) 0 ['block3a_activation[0][0]']
AveragePooling2D)
block3a_se_reshape (Reshap (None, 1, 1, 144) 0 ['block3a_se_squeeze[0][0]']
e)
block3a_se_reduce (Conv2D) (None, 1, 1, 6) 870 ['block3a_se_reshape[0][0]']
block3a_se_expand (Conv2D) (None, 1, 1, 144) 1008 ['block3a_se_reduce[0][0]']
block3a_se_excite (Multipl (None, 28, 28, 144) 0 ['block3a_activation[0][0]',
y) 'block3a_se_expand[0][0]']
block3a_project_conv (Conv (None, 28, 28, 40) 5760 ['block3a_se_excite[0][0]']
2D)
block3a_project_bn (BatchN (None, 28, 28, 40) 160 ['block3a_project_conv[0][0]']
ormalization)
block3b_expand_conv (Conv2 (None, 28, 28, 240) 9600 ['block3a_project_bn[0][0]']
D)
block3b_expand_bn (BatchNo (None, 28, 28, 240) 960 ['block3b_expand_conv[0][0]']
rmalization)
block3b_expand_activation (None, 28, 28, 240) 0 ['block3b_expand_bn[0][0]']
(Activation)
block3b_dwconv (DepthwiseC (None, 28, 28, 240) 6000 ['block3b_expand_activation[0]
onv2D) [0]']
block3b_bn (BatchNormaliza (None, 28, 28, 240) 960 ['block3b_dwconv[0][0]']
tion)
block3b_activation (Activa (None, 28, 28, 240) 0 ['block3b_bn[0][0]']
tion)
block3b_se_squeeze (Global (None, 240) 0 ['block3b_activation[0][0]']
AveragePooling2D)
block3b_se_reshape (Reshap (None, 1, 1, 240) 0 ['block3b_se_squeeze[0][0]']
e)
block3b_se_reduce (Conv2D) (None, 1, 1, 10) 2410 ['block3b_se_reshape[0][0]']
block3b_se_expand (Conv2D) (None, 1, 1, 240) 2640 ['block3b_se_reduce[0][0]']
block3b_se_excite (Multipl (None, 28, 28, 240) 0 ['block3b_activation[0][0]',
y) 'block3b_se_expand[0][0]']
block3b_project_conv (Conv (None, 28, 28, 40) 9600 ['block3b_se_excite[0][0]']
2D)
block3b_project_bn (BatchN (None, 28, 28, 40) 160 ['block3b_project_conv[0][0]']
ormalization)
block3b_drop (FixedDropout (None, 28, 28, 40) 0 ['block3b_project_bn[0][0]']
)
block3b_add (Add) (None, 28, 28, 40) 0 ['block3b_drop[0][0]',
'block3a_project_bn[0][0]']
block4a_expand_conv (Conv2 (None, 28, 28, 240) 9600 ['block3b_add[0][0]']
D)
block4a_expand_bn (BatchNo (None, 28, 28, 240) 960 ['block4a_expand_conv[0][0]']
rmalization)
block4a_expand_activation (None, 28, 28, 240) 0 ['block4a_expand_bn[0][0]']
(Activation)
block4a_dwconv (DepthwiseC (None, 14, 14, 240) 2160 ['block4a_expand_activation[0]
onv2D) [0]']
block4a_bn (BatchNormaliza (None, 14, 14, 240) 960 ['block4a_dwconv[0][0]']
tion)
block4a_activation (Activa (None, 14, 14, 240) 0 ['block4a_bn[0][0]']
tion)
block4a_se_squeeze (Global (None, 240) 0 ['block4a_activation[0][0]']
AveragePooling2D)
block4a_se_reshape (Reshap (None, 1, 1, 240) 0 ['block4a_se_squeeze[0][0]']
e)
block4a_se_reduce (Conv2D) (None, 1, 1, 10) 2410 ['block4a_se_reshape[0][0]']
block4a_se_expand (Conv2D) (None, 1, 1, 240) 2640 ['block4a_se_reduce[0][0]']
block4a_se_excite (Multipl (None, 14, 14, 240) 0 ['block4a_activation[0][0]',
y) 'block4a_se_expand[0][0]']
block4a_project_conv (Conv (None, 14, 14, 80) 19200 ['block4a_se_excite[0][0]']
2D)
block4a_project_bn (BatchN (None, 14, 14, 80) 320 ['block4a_project_conv[0][0]']
ormalization)
block4b_expand_conv (Conv2 (None, 14, 14, 480) 38400 ['block4a_project_bn[0][0]']
D)
block4b_expand_bn (BatchNo (None, 14, 14, 480) 1920 ['block4b_expand_conv[0][0]']
rmalization)
block4b_expand_activation (None, 14, 14, 480) 0 ['block4b_expand_bn[0][0]']
(Activation)
block4b_dwconv (DepthwiseC (None, 14, 14, 480) 4320 ['block4b_expand_activation[0]
onv2D) [0]']
block4b_bn (BatchNormaliza (None, 14, 14, 480) 1920 ['block4b_dwconv[0][0]']
tion)
block4b_activation (Activa (None, 14, 14, 480) 0 ['block4b_bn[0][0]']
tion)
block4b_se_squeeze (Global (None, 480) 0 ['block4b_activation[0][0]']
AveragePooling2D)
block4b_se_reshape (Reshap (None, 1, 1, 480) 0 ['block4b_se_squeeze[0][0]']
e)
block4b_se_reduce (Conv2D) (None, 1, 1, 20) 9620 ['block4b_se_reshape[0][0]']
block4b_se_expand (Conv2D) (None, 1, 1, 480) 10080 ['block4b_se_reduce[0][0]']
block4b_se_excite (Multipl (None, 14, 14, 480) 0 ['block4b_activation[0][0]',
y) 'block4b_se_expand[0][0]']
block4b_project_conv (Conv (None, 14, 14, 80) 38400 ['block4b_se_excite[0][0]']
2D)
block4b_project_bn (BatchN (None, 14, 14, 80) 320 ['block4b_project_conv[0][0]']
ormalization)
block4b_drop (FixedDropout (None, 14, 14, 80) 0 ['block4b_project_bn[0][0]']
)
block4b_add (Add) (None, 14, 14, 80) 0 ['block4b_drop[0][0]',
'block4a_project_bn[0][0]']
block4c_expand_conv (Conv2 (None, 14, 14, 480) 38400 ['block4b_add[0][0]']
D)
block4c_expand_bn (BatchNo (None, 14, 14, 480) 1920 ['block4c_expand_conv[0][0]']
rmalization)
block4c_expand_activation (None, 14, 14, 480) 0 ['block4c_expand_bn[0][0]']
(Activation)
block4c_dwconv (DepthwiseC (None, 14, 14, 480) 4320 ['block4c_expand_activation[0]
onv2D) [0]']
block4c_bn (BatchNormaliza (None, 14, 14, 480) 1920 ['block4c_dwconv[0][0]']
tion)
block4c_activation (Activa (None, 14, 14, 480) 0 ['block4c_bn[0][0]']
tion)
block4c_se_squeeze (Global (None, 480) 0 ['block4c_activation[0][0]']
AveragePooling2D)
block4c_se_reshape (Reshap (None, 1, 1, 480) 0 ['block4c_se_squeeze[0][0]']
e)
block4c_se_reduce (Conv2D) (None, 1, 1, 20) 9620 ['block4c_se_reshape[0][0]']
block4c_se_expand (Conv2D) (None, 1, 1, 480) 10080 ['block4c_se_reduce[0][0]']
block4c_se_excite (Multipl (None, 14, 14, 480) 0 ['block4c_activation[0][0]',
y) 'block4c_se_expand[0][0]']
block4c_project_conv (Conv (None, 14, 14, 80) 38400 ['block4c_se_excite[0][0]']
2D)
block4c_project_bn (BatchN (None, 14, 14, 80) 320 ['block4c_project_conv[0][0]']
ormalization)
block4c_drop (FixedDropout (None, 14, 14, 80) 0 ['block4c_project_bn[0][0]']
)
block4c_add (Add) (None, 14, 14, 80) 0 ['block4c_drop[0][0]',
'block4b_add[0][0]']
block5a_expand_conv (Conv2 (None, 14, 14, 480) 38400 ['block4c_add[0][0]']
D)
block5a_expand_bn (BatchNo (None, 14, 14, 480) 1920 ['block5a_expand_conv[0][0]']
rmalization)
block5a_expand_activation (None, 14, 14, 480) 0 ['block5a_expand_bn[0][0]']
(Activation)
block5a_dwconv (DepthwiseC (None, 14, 14, 480) 12000 ['block5a_expand_activation[0]
onv2D) [0]']
block5a_bn (BatchNormaliza (None, 14, 14, 480) 1920 ['block5a_dwconv[0][0]']
tion)
block5a_activation (Activa (None, 14, 14, 480) 0 ['block5a_bn[0][0]']
tion)
block5a_se_squeeze (Global (None, 480) 0 ['block5a_activation[0][0]']
AveragePooling2D)
block5a_se_reshape (Reshap (None, 1, 1, 480) 0 ['block5a_se_squeeze[0][0]']
e)
block5a_se_reduce (Conv2D) (None, 1, 1, 20) 9620 ['block5a_se_reshape[0][0]']
block5a_se_expand (Conv2D) (None, 1, 1, 480) 10080 ['block5a_se_reduce[0][0]']
block5a_se_excite (Multipl (None, 14, 14, 480) 0 ['block5a_activation[0][0]',
y) 'block5a_se_expand[0][0]']
block5a_project_conv (Conv (None, 14, 14, 112) 53760 ['block5a_se_excite[0][0]']
2D)
block5a_project_bn (BatchN (None, 14, 14, 112) 448 ['block5a_project_conv[0][0]']
ormalization)
block5b_expand_conv (Conv2 (None, 14, 14, 672) 75264 ['block5a_project_bn[0][0]']
D)
block5b_expand_bn (BatchNo (None, 14, 14, 672) 2688 ['block5b_expand_conv[0][0]']
rmalization)
block5b_expand_activation (None, 14, 14, 672) 0 ['block5b_expand_bn[0][0]']
(Activation)
block5b_dwconv (DepthwiseC (None, 14, 14, 672) 16800 ['block5b_expand_activation[0]
onv2D) [0]']
block5b_bn (BatchNormaliza (None, 14, 14, 672) 2688 ['block5b_dwconv[0][0]']
tion)
block5b_activation (Activa (None, 14, 14, 672) 0 ['block5b_bn[0][0]']
tion)
block5b_se_squeeze (Global (None, 672) 0 ['block5b_activation[0][0]']
AveragePooling2D)
block5b_se_reshape (Reshap (None, 1, 1, 672) 0 ['block5b_se_squeeze[0][0]']
e)
block5b_se_reduce (Conv2D) (None, 1, 1, 28) 18844 ['block5b_se_reshape[0][0]']
block5b_se_expand (Conv2D) (None, 1, 1, 672) 19488 ['block5b_se_reduce[0][0]']
block5b_se_excite (Multipl (None, 14, 14, 672) 0 ['block5b_activation[0][0]',
y) 'block5b_se_expand[0][0]']
block5b_project_conv (Conv (None, 14, 14, 112) 75264 ['block5b_se_excite[0][0]']
2D)
block5b_project_bn (BatchN (None, 14, 14, 112) 448 ['block5b_project_conv[0][0]']
ormalization)
block5b_drop (FixedDropout (None, 14, 14, 112) 0 ['block5b_project_bn[0][0]']
)
block5b_add (Add) (None, 14, 14, 112) 0 ['block5b_drop[0][0]',
'block5a_project_bn[0][0]']
block5c_expand_conv (Conv2 (None, 14, 14, 672) 75264 ['block5b_add[0][0]']
D)
block5c_expand_bn (BatchNo (None, 14, 14, 672) 2688 ['block5c_expand_conv[0][0]']
rmalization)
block5c_expand_activation (None, 14, 14, 672) 0 ['block5c_expand_bn[0][0]']
(Activation)
block5c_dwconv (DepthwiseC (None, 14, 14, 672) 16800 ['block5c_expand_activation[0]
onv2D) [0]']
block5c_bn (BatchNormaliza (None, 14, 14, 672) 2688 ['block5c_dwconv[0][0]']
tion)
block5c_activation (Activa (None, 14, 14, 672) 0 ['block5c_bn[0][0]']
tion)
block5c_se_squeeze (Global (None, 672) 0 ['block5c_activation[0][0]']
AveragePooling2D)
block5c_se_reshape (Reshap (None, 1, 1, 672) 0 ['block5c_se_squeeze[0][0]']
e)
block5c_se_reduce (Conv2D) (None, 1, 1, 28) 18844 ['block5c_se_reshape[0][0]']
block5c_se_expand (Conv2D) (None, 1, 1, 672) 19488 ['block5c_se_reduce[0][0]']
block5c_se_excite (Multipl (None, 14, 14, 672) 0 ['block5c_activation[0][0]',
y) 'block5c_se_expand[0][0]']
block5c_project_conv (Conv (None, 14, 14, 112) 75264 ['block5c_se_excite[0][0]']
2D)
block5c_project_bn (BatchN (None, 14, 14, 112) 448 ['block5c_project_conv[0][0]']
ormalization)
block5c_drop (FixedDropout (None, 14, 14, 112) 0 ['block5c_project_bn[0][0]']
)
block5c_add (Add) (None, 14, 14, 112) 0 ['block5c_drop[0][0]',
'block5b_add[0][0]']
block6a_expand_conv (Conv2 (None, 14, 14, 672) 75264 ['block5c_add[0][0]']
D)
block6a_expand_bn (BatchNo (None, 14, 14, 672) 2688 ['block6a_expand_conv[0][0]']
rmalization)
block6a_expand_activation (None, 14, 14, 672) 0 ['block6a_expand_bn[0][0]']
(Activation)
block6a_dwconv (DepthwiseC (None, 7, 7, 672) 16800 ['block6a_expand_activation[0]
onv2D) [0]']
block6a_bn (BatchNormaliza (None, 7, 7, 672) 2688 ['block6a_dwconv[0][0]']
tion)
block6a_activation (Activa (None, 7, 7, 672) 0 ['block6a_bn[0][0]']
tion)
block6a_se_squeeze (Global (None, 672) 0 ['block6a_activation[0][0]']
AveragePooling2D)
block6a_se_reshape (Reshap (None, 1, 1, 672) 0 ['block6a_se_squeeze[0][0]']
e)
block6a_se_reduce (Conv2D) (None, 1, 1, 28) 18844 ['block6a_se_reshape[0][0]']
block6a_se_expand (Conv2D) (None, 1, 1, 672) 19488 ['block6a_se_reduce[0][0]']
block6a_se_excite (Multipl (None, 7, 7, 672) 0 ['block6a_activation[0][0]',
y) 'block6a_se_expand[0][0]']
block6a_project_conv (Conv (None, 7, 7, 192) 129024 ['block6a_se_excite[0][0]']
2D)
block6a_project_bn (BatchN (None, 7, 7, 192) 768 ['block6a_project_conv[0][0]']
ormalization)
block6b_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6a_project_bn[0][0]']
D)
block6b_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block6b_expand_conv[0][0]']
rmalization)
block6b_expand_activation (None, 7, 7, 1152) 0 ['block6b_expand_bn[0][0]']
(Activation)
block6b_dwconv (DepthwiseC (None, 7, 7, 1152) 28800 ['block6b_expand_activation[0]
onv2D) [0]']
block6b_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block6b_dwconv[0][0]']
tion)
block6b_activation (Activa (None, 7, 7, 1152) 0 ['block6b_bn[0][0]']
tion)
block6b_se_squeeze (Global (None, 1152) 0 ['block6b_activation[0][0]']
AveragePooling2D)
block6b_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block6b_se_squeeze[0][0]']
e)
block6b_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block6b_se_reshape[0][0]']
block6b_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block6b_se_reduce[0][0]']
block6b_se_excite (Multipl (None, 7, 7, 1152) 0 ['block6b_activation[0][0]',
y) 'block6b_se_expand[0][0]']
block6b_project_conv (Conv (None, 7, 7, 192) 221184 ['block6b_se_excite[0][0]']
2D)
block6b_project_bn (BatchN (None, 7, 7, 192) 768 ['block6b_project_conv[0][0]']
ormalization)
block6b_drop (FixedDropout (None, 7, 7, 192) 0 ['block6b_project_bn[0][0]']
)
block6b_add (Add) (None, 7, 7, 192) 0 ['block6b_drop[0][0]',
'block6a_project_bn[0][0]']
block6c_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6b_add[0][0]']
D)
block6c_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block6c_expand_conv[0][0]']
rmalization)
block6c_expand_activation (None, 7, 7, 1152) 0 ['block6c_expand_bn[0][0]']
(Activation)
block6c_dwconv (DepthwiseC (None, 7, 7, 1152) 28800 ['block6c_expand_activation[0]
onv2D) [0]']
block6c_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block6c_dwconv[0][0]']
tion)
block6c_activation (Activa (None, 7, 7, 1152) 0 ['block6c_bn[0][0]']
tion)
block6c_se_squeeze (Global (None, 1152) 0 ['block6c_activation[0][0]']
AveragePooling2D)
block6c_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block6c_se_squeeze[0][0]']
e)
block6c_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block6c_se_reshape[0][0]']
block6c_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block6c_se_reduce[0][0]']
block6c_se_excite (Multipl (None, 7, 7, 1152) 0 ['block6c_activation[0][0]',
y) 'block6c_se_expand[0][0]']
block6c_project_conv (Conv (None, 7, 7, 192) 221184 ['block6c_se_excite[0][0]']
2D)
block6c_project_bn (BatchN (None, 7, 7, 192) 768 ['block6c_project_conv[0][0]']
ormalization)
block6c_drop (FixedDropout (None, 7, 7, 192) 0 ['block6c_project_bn[0][0]']
)
block6c_add (Add) (None, 7, 7, 192) 0 ['block6c_drop[0][0]',
'block6b_add[0][0]']
block6d_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6c_add[0][0]']
D)
block6d_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block6d_expand_conv[0][0]']
rmalization)
block6d_expand_activation (None, 7, 7, 1152) 0 ['block6d_expand_bn[0][0]']
(Activation)
block6d_dwconv (DepthwiseC (None, 7, 7, 1152) 28800 ['block6d_expand_activation[0]
onv2D) [0]']
block6d_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block6d_dwconv[0][0]']
tion)
block6d_activation (Activa (None, 7, 7, 1152) 0 ['block6d_bn[0][0]']
tion)
block6d_se_squeeze (Global (None, 1152) 0 ['block6d_activation[0][0]']
AveragePooling2D)
block6d_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block6d_se_squeeze[0][0]']
e)
block6d_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block6d_se_reshape[0][0]']
block6d_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block6d_se_reduce[0][0]']
block6d_se_excite (Multipl (None, 7, 7, 1152) 0 ['block6d_activation[0][0]',
y) 'block6d_se_expand[0][0]']
block6d_project_conv (Conv (None, 7, 7, 192) 221184 ['block6d_se_excite[0][0]']
2D)
block6d_project_bn (BatchN (None, 7, 7, 192) 768 ['block6d_project_conv[0][0]']
ormalization)
block6d_drop (FixedDropout (None, 7, 7, 192) 0 ['block6d_project_bn[0][0]']
)
block6d_add (Add) (None, 7, 7, 192) 0 ['block6d_drop[0][0]',
'block6c_add[0][0]']
block7a_expand_conv (Conv2 (None, 7, 7, 1152) 221184 ['block6d_add[0][0]']
D)
block7a_expand_bn (BatchNo (None, 7, 7, 1152) 4608 ['block7a_expand_conv[0][0]']
rmalization)
block7a_expand_activation (None, 7, 7, 1152) 0 ['block7a_expand_bn[0][0]']
(Activation)
block7a_dwconv (DepthwiseC (None, 7, 7, 1152) 10368 ['block7a_expand_activation[0]
onv2D) [0]']
block7a_bn (BatchNormaliza (None, 7, 7, 1152) 4608 ['block7a_dwconv[0][0]']
tion)
block7a_activation (Activa (None, 7, 7, 1152) 0 ['block7a_bn[0][0]']
tion)
block7a_se_squeeze (Global (None, 1152) 0 ['block7a_activation[0][0]']
AveragePooling2D)
block7a_se_reshape (Reshap (None, 1, 1, 1152) 0 ['block7a_se_squeeze[0][0]']
e)
block7a_se_reduce (Conv2D) (None, 1, 1, 48) 55344 ['block7a_se_reshape[0][0]']
block7a_se_expand (Conv2D) (None, 1, 1, 1152) 56448 ['block7a_se_reduce[0][0]']
block7a_se_excite (Multipl (None, 7, 7, 1152) 0 ['block7a_activation[0][0]',
y) 'block7a_se_expand[0][0]']
block7a_project_conv (Conv (None, 7, 7, 320) 368640 ['block7a_se_excite[0][0]']
2D)
block7a_project_bn (BatchN (None, 7, 7, 320) 1280 ['block7a_project_conv[0][0]']
ormalization)
top_conv (Conv2D) (None, 7, 7, 1280) 409600 ['block7a_project_bn[0][0]']
top_bn (BatchNormalization (None, 7, 7, 1280) 5120 ['top_conv[0][0]']
)
top_activation (Activation (None, 7, 7, 1280) 0 ['top_bn[0][0]']
)
avg_pool (GlobalAveragePoo (None, 1280) 0 ['top_activation[0][0]']
ling2D)
top_dropout (Dropout) (None, 1280) 0 ['avg_pool[0][0]']
==================================================================================================
Total params: 4049564 (15.45 MB)
Trainable params: 4007548 (15.29 MB)
Non-trainable params: 42016 (164.12 KB)
__________________________________________________________________________________________________
# # Making all the layers of the Resnet model non-trainable. i.e. freezing them
for layer in effnet_model.layers:
layer.trainable = False
print(layer.name, layer.trainable)
input_1 False stem_conv False stem_bn False stem_activation False block1a_dwconv False block1a_bn False block1a_activation False block1a_se_squeeze False block1a_se_reshape False block1a_se_reduce False block1a_se_expand False block1a_se_excite False block1a_project_conv False block1a_project_bn False block2a_expand_conv False block2a_expand_bn False block2a_expand_activation False block2a_dwconv False block2a_bn False block2a_activation False block2a_se_squeeze False block2a_se_reshape False block2a_se_reduce False block2a_se_expand False block2a_se_excite False block2a_project_conv False block2a_project_bn False block2b_expand_conv False block2b_expand_bn False block2b_expand_activation False block2b_dwconv False block2b_bn False block2b_activation False block2b_se_squeeze False block2b_se_reshape False block2b_se_reduce False block2b_se_expand False block2b_se_excite False block2b_project_conv False block2b_project_bn False block2b_drop False block2b_add False block3a_expand_conv False block3a_expand_bn False block3a_expand_activation False block3a_dwconv False block3a_bn False block3a_activation False block3a_se_squeeze False block3a_se_reshape False block3a_se_reduce False block3a_se_expand False block3a_se_excite False block3a_project_conv False block3a_project_bn False block3b_expand_conv False block3b_expand_bn False block3b_expand_activation False block3b_dwconv False block3b_bn False block3b_activation False block3b_se_squeeze False block3b_se_reshape False block3b_se_reduce False block3b_se_expand False block3b_se_excite False block3b_project_conv False block3b_project_bn False block3b_drop False block3b_add False block4a_expand_conv False block4a_expand_bn False block4a_expand_activation False block4a_dwconv False block4a_bn False block4a_activation False block4a_se_squeeze False block4a_se_reshape False block4a_se_reduce False block4a_se_expand False block4a_se_excite False block4a_project_conv False block4a_project_bn False block4b_expand_conv False block4b_expand_bn False block4b_expand_activation False block4b_dwconv False block4b_bn False block4b_activation False block4b_se_squeeze False block4b_se_reshape False block4b_se_reduce False block4b_se_expand False block4b_se_excite False block4b_project_conv False block4b_project_bn False block4b_drop False block4b_add False block4c_expand_conv False block4c_expand_bn False block4c_expand_activation False block4c_dwconv False block4c_bn False block4c_activation False block4c_se_squeeze False block4c_se_reshape False block4c_se_reduce False block4c_se_expand False block4c_se_excite False block4c_project_conv False block4c_project_bn False block4c_drop False block4c_add False block5a_expand_conv False block5a_expand_bn False block5a_expand_activation False block5a_dwconv False block5a_bn False block5a_activation False block5a_se_squeeze False block5a_se_reshape False block5a_se_reduce False block5a_se_expand False block5a_se_excite False block5a_project_conv False block5a_project_bn False block5b_expand_conv False block5b_expand_bn False block5b_expand_activation False block5b_dwconv False block5b_bn False block5b_activation False block5b_se_squeeze False block5b_se_reshape False block5b_se_reduce False block5b_se_expand False block5b_se_excite False block5b_project_conv False block5b_project_bn False block5b_drop False block5b_add False block5c_expand_conv False block5c_expand_bn False block5c_expand_activation False block5c_dwconv False block5c_bn False block5c_activation False block5c_se_squeeze False block5c_se_reshape False block5c_se_reduce False block5c_se_expand False block5c_se_excite False block5c_project_conv False block5c_project_bn False block5c_drop False block5c_add False block6a_expand_conv False block6a_expand_bn False block6a_expand_activation False block6a_dwconv False block6a_bn False block6a_activation False block6a_se_squeeze False block6a_se_reshape False block6a_se_reduce False block6a_se_expand False block6a_se_excite False block6a_project_conv False block6a_project_bn False block6b_expand_conv False block6b_expand_bn False block6b_expand_activation False block6b_dwconv False block6b_bn False block6b_activation False block6b_se_squeeze False block6b_se_reshape False block6b_se_reduce False block6b_se_expand False block6b_se_excite False block6b_project_conv False block6b_project_bn False block6b_drop False block6b_add False block6c_expand_conv False block6c_expand_bn False block6c_expand_activation False block6c_dwconv False block6c_bn False block6c_activation False block6c_se_squeeze False block6c_se_reshape False block6c_se_reduce False block6c_se_expand False block6c_se_excite False block6c_project_conv False block6c_project_bn False block6c_drop False block6c_add False block6d_expand_conv False block6d_expand_bn False block6d_expand_activation False block6d_dwconv False block6d_bn False block6d_activation False block6d_se_squeeze False block6d_se_reshape False block6d_se_reduce False block6d_se_expand False block6d_se_excite False block6d_project_conv False block6d_project_bn False block6d_drop False block6d_add False block7a_expand_conv False block7a_expand_bn False block7a_expand_activation False block7a_dwconv False block7a_bn False block7a_activation False block7a_se_squeeze False block7a_se_reshape False block7a_se_reduce False block7a_se_expand False block7a_se_excite False block7a_project_conv False block7a_project_bn False top_conv False top_bn False top_activation False avg_pool False top_dropout False
# building CNN model with Efficient Net engine
# I will use same CNN structure and only replace the engine to Efficient Net
effnet_engine_model = Sequential()
# Adding the convolutional part of the Efficient Net created above
effnet_engine_model.add(resnet_model)
# Flattening the output of the Efficient Net engine
effnet_engine_model.add(Flatten())
# Adding a dense output layer
effnet_engine_model.add(Dense(32))
effnet_engine_model.add(LeakyReLU(0.1))
effnet_engine_model.add(Dropout(0.1))
effnet_engine_model.add(Dense(32))
effnet_engine_model.add(LeakyReLU(0.1))
effnet_engine_model.add(Dense(16))
effnet_engine_model.add(LeakyReLU(0.1))
effnet_engine_model.add(Dense(4, activation='softmax'))
# print model summary
effnet_engine_model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
model (Functional) (None, 2048) 23564800
flatten (Flatten) (None, 2048) 0
dense (Dense) (None, 32) 65568
leaky_re_lu (LeakyReLU) (None, 32) 0
dropout (Dropout) (None, 32) 0
dense_1 (Dense) (None, 32) 1056
leaky_re_lu_1 (LeakyReLU) (None, 32) 0
dense_2 (Dense) (None, 16) 528
leaky_re_lu_2 (LeakyReLU) (None, 16) 0
dense_3 (Dense) (None, 4) 68
=================================================================
Total params: 23632020 (90.15 MB)
Trainable params: 67220 (262.58 KB)
Non-trainable params: 23564800 (89.89 MB)
_________________________________________________________________
# compiling the model with Adam optimizer, categorical crossentropy loss function as this is multi class model
# and accuracy metrics to evaluate the models performance during traiing and validation
opt = Adam(learning_rate = 0.001) #use Adam optimizer with default learning rate
effnet_engine_model.compile(loss="categorical_crossentropy", optimizer=opt, metrics = ['accuracy'])
# Pulling a single large batch of random validatio data for validating after each epoch
valX, valY = validation_generator_rgb_2.next()
# the following code will stop the training if the val_loss continues above min val_loss for the run for more than 8 epochs
es = EarlyStopping(monitor = 'val_loss', mode = 'min', verbose = 1, patience = 8)
# Reduce LR on Plateau callback will reduce the learning rate to 0.2 of its current value, if val_loss does not improve for 5 consecutive epochs
learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_loss',
patience = 5,
verbose = 1,
factor = 0.2,
cooldown = 2,
min_lr = 0.0001)
effnet_hist = effnet_engine_model.fit(train_generator_rgb_2,
validation_data=(valX, valY),
epochs=20,
callbacks = [es, learning_rate_reduction])
Epoch 1/20 237/237 [==============================] - 39s 149ms/step - loss: 1.0260 - accuracy: 0.5628 - val_loss: 0.7990 - val_accuracy: 0.6875 - lr: 0.0010 Epoch 2/20 237/237 [==============================] - 32s 134ms/step - loss: 0.8473 - accuracy: 0.6492 - val_loss: 0.7924 - val_accuracy: 0.6953 - lr: 0.0010 Epoch 3/20 237/237 [==============================] - 25s 107ms/step - loss: 0.7877 - accuracy: 0.6758 - val_loss: 0.7204 - val_accuracy: 0.7266 - lr: 0.0010 Epoch 4/20 237/237 [==============================] - 31s 130ms/step - loss: 0.7404 - accuracy: 0.6990 - val_loss: 0.7385 - val_accuracy: 0.7031 - lr: 0.0010 Epoch 5/20 237/237 [==============================] - 26s 108ms/step - loss: 0.7030 - accuracy: 0.7183 - val_loss: 0.7082 - val_accuracy: 0.7188 - lr: 0.0010 Epoch 6/20 237/237 [==============================] - 36s 151ms/step - loss: 0.6713 - accuracy: 0.7285 - val_loss: 0.7536 - val_accuracy: 0.6719 - lr: 0.0010 Epoch 7/20 237/237 [==============================] - 33s 137ms/step - loss: 0.6423 - accuracy: 0.7376 - val_loss: 0.8407 - val_accuracy: 0.6641 - lr: 0.0010 Epoch 8/20 237/237 [==============================] - 28s 119ms/step - loss: 0.6178 - accuracy: 0.7462 - val_loss: 0.7137 - val_accuracy: 0.6953 - lr: 0.0010 Epoch 9/20 237/237 [==============================] - 34s 144ms/step - loss: 0.5906 - accuracy: 0.7619 - val_loss: 0.6687 - val_accuracy: 0.7344 - lr: 0.0010 Epoch 10/20 237/237 [==============================] - 27s 112ms/step - loss: 0.5623 - accuracy: 0.7729 - val_loss: 0.7494 - val_accuracy: 0.7188 - lr: 0.0010 Epoch 11/20 237/237 [==============================] - 28s 119ms/step - loss: 0.5358 - accuracy: 0.7848 - val_loss: 0.9074 - val_accuracy: 0.6797 - lr: 0.0010 Epoch 12/20 237/237 [==============================] - 30s 127ms/step - loss: 0.5313 - accuracy: 0.7844 - val_loss: 0.9001 - val_accuracy: 0.6797 - lr: 0.0010 Epoch 13/20 237/237 [==============================] - 28s 118ms/step - loss: 0.5090 - accuracy: 0.7985 - val_loss: 0.8636 - val_accuracy: 0.7031 - lr: 0.0010 Epoch 14/20 237/237 [==============================] - ETA: 0s - loss: 0.4958 - accuracy: 0.7993 Epoch 14: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 237/237 [==============================] - 33s 140ms/step - loss: 0.4958 - accuracy: 0.7993 - val_loss: 0.8216 - val_accuracy: 0.7109 - lr: 0.0010 Epoch 15/20 237/237 [==============================] - 29s 121ms/step - loss: 0.4301 - accuracy: 0.8310 - val_loss: 0.8594 - val_accuracy: 0.7031 - lr: 2.0000e-04 Epoch 16/20 237/237 [==============================] - 34s 144ms/step - loss: 0.4049 - accuracy: 0.8415 - val_loss: 0.8559 - val_accuracy: 0.6953 - lr: 2.0000e-04 Epoch 17/20 237/237 [==============================] - 34s 143ms/step - loss: 0.3950 - accuracy: 0.8463 - val_loss: 0.8817 - val_accuracy: 0.7031 - lr: 2.0000e-04 Epoch 17: early stopping
# evaluate vgg_engine_model on test data
effnet_engine_model.evaluate(test_generator_rgb_2)
1/1 [==============================] - 0s 372ms/step - loss: 0.8468 - accuracy: 0.7031
[0.8468083143234253, 0.703125]
plot_accuracy(effnet_hist)
plot_loss(effnet_hist)
# get test images and their classes
test_images, test_classes = next(test_generator_rgb_2)
actual_classes = np.argmax(test_classes, axis = -1)
# Make predictions
effnet_predictions = effnet_engine_model.predict(test_images)
# Convert predictions to class labels
effnet_predictions = np.argmax(effnet_predictions, axis = -1)
# generate classification report and confussion matrix
metrics_score(actual_classes,effnet_predictions )
4/4 [==============================] - 0s 33ms/step
precision recall f1-score support
0 0.76 0.78 0.77 32
1 0.53 0.50 0.52 32
2 0.67 0.75 0.71 32
3 0.86 0.78 0.82 32
accuracy 0.70 128
macro avg 0.70 0.70 0.70 128
weighted avg 0.70 0.70 0.70 128
Observations and Insights:
Note: You can even go back and build your own architecture on top of the VGG16 Transfer layer and see if you can improve the performance.
Think About It:
Now that we have tried multiple pre-trained models, let's build a complex CNN architecture and see if we can get better performance.
In this section, we will build a more complex Convolutional Neural Network Model that has close to as many parameters as we had in our Transfer Learning Models. However, we will have only 1 input channel for our input images.
In this section, we are creating data loaders which we will use as inputs to the more Complicated Convolutional Neural Network. We will go ahead with color_mode = 'grayscale'.
# clear the previous model's history from the Keras backend
from tensorflow.keras import backend
backend.clear_session()
# Fixing the seed for random number generators so that we can ensure we receive the same output everytime
np.random.seed(666)
import random
random.seed(666)
tf.random.set_seed(666)
# Flowing training images in batches of 20 using train_datagen generator grayscale
train_generator_gray_2 = train_datagen.flow_from_directory(train_dir,
batch_size=64,
class_mode='categorical',
target_size=(48, 48),
color_mode = 'grayscale')
# Flowing testing images in batches of 20 using test_datagen generator grayscale
test_generator_gray_2 = test_datagen.flow_from_directory(test_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (48, 48),
color_mode = 'grayscale')
# Flowing validation images in batches of 20 using test_datagen generator grayscale
validation_generator_gray_2 = validation_datagen.flow_from_directory(val_dir,
batch_size=128,
class_mode = 'categorical',
target_size = (48, 48),
color_mode = 'grayscale')
Found 15109 images belonging to 4 classes. Found 128 images belonging to 4 classes. Found 4977 images belonging to 4 classes.
# CNN composed of 5 convolutional blocks
cnn_model_3 = Sequential()
cnn_model_3.add(Conv2D(256, (3,3), input_shape=(48, 48, 1), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(Conv2D(256, (3,3), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(MaxPooling2D(2,2))
cnn_model_3.add(BatchNormalization())
cnn_model_3.add(Conv2D(128, (3,3), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(Conv2D(128, (3,3), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(MaxPooling2D(2,2))
cnn_model_3.add(BatchNormalization())
cnn_model_3.add(Conv2D(64, (3,3), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(Conv2D(64, (3,3), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(MaxPooling2D(2,2))
cnn_model_3.add(Conv2D(32, (3,3), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(MaxPooling2D(2,2))
cnn_model_3.add(BatchNormalization())
cnn_model_3.add(Conv2D(16, (3,3), padding = 'same'))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(BatchNormalization())
cnn_model_3.add(Flatten())
cnn_model_3.add(Dense(32))
cnn_model_3.add(LeakyReLU(0.1))
cnn_model_3.add(Dropout(0.2))
cnn_model_3.add(Dense(32))
cnn_model_3.add(LeakyReLU(0.2))
cnn_model_3.add(Dense(32))
cnn_model_3.add(LeakyReLU(0.2))
cnn_model_3.add(Dense(4, activation='softmax'))
# compiling the model with Adam optimizer, categorical crossentropy loss function as this is multi class model
# and accuracy metrics to evaluate the models performance during traiing and validation
opt = Adam(learning_rate = 0.001) #use Adam optimizer with default learning rate
cnn_model_3.compile(loss="categorical_crossentropy", optimizer=opt, metrics = ['accuracy'])
# Pulling a single large batch of random validatio data for validating after each epoch
valX, valY = validation_generator_gray_2.next()
# the following code will stop the training if the val_loss continues above min val_loss for the run for more than 8 epochs
es = EarlyStopping(monitor = 'val_loss', mode = 'min', verbose = 1, patience = 8)
# Reduce LR on Plateau callback will reduce the learning rate to 0.2 of its current value, if val_loss does not improve for 5 consecutive epochs
learning_rate_reduction = ReduceLROnPlateau(monitor = 'val_loss',
patience = 5,
verbose = 1,
factor = 0.2,
cooldown = 2,
min_lr = 0.0001)
cnn_3_hist = cnn_model_3.fit(train_generator_gray_2,
validation_data=(valX, valY),
epochs=20,
callbacks = [es, learning_rate_reduction])
Epoch 1/20 237/237 [==============================] - 14s 41ms/step - loss: 1.2577 - accuracy: 0.3983 - val_loss: 1.2567 - val_accuracy: 0.4375 - lr: 0.0010 Epoch 2/20 237/237 [==============================] - 9s 40ms/step - loss: 0.9883 - accuracy: 0.5624 - val_loss: 0.9693 - val_accuracy: 0.6172 - lr: 0.0010 Epoch 3/20 237/237 [==============================] - 11s 46ms/step - loss: 0.8263 - accuracy: 0.6480 - val_loss: 1.1891 - val_accuracy: 0.4453 - lr: 0.0010 Epoch 4/20 237/237 [==============================] - 8s 35ms/step - loss: 0.7372 - accuracy: 0.6903 - val_loss: 0.7970 - val_accuracy: 0.6484 - lr: 0.0010 Epoch 5/20 237/237 [==============================] - 10s 40ms/step - loss: 0.6718 - accuracy: 0.7289 - val_loss: 0.8621 - val_accuracy: 0.6641 - lr: 0.0010 Epoch 6/20 237/237 [==============================] - 10s 40ms/step - loss: 0.6218 - accuracy: 0.7466 - val_loss: 0.8477 - val_accuracy: 0.6406 - lr: 0.0010 Epoch 7/20 237/237 [==============================] - 8s 35ms/step - loss: 0.5687 - accuracy: 0.7658 - val_loss: 0.8193 - val_accuracy: 0.6797 - lr: 0.0010 Epoch 8/20 237/237 [==============================] - 8s 35ms/step - loss: 0.5320 - accuracy: 0.7844 - val_loss: 0.7260 - val_accuracy: 0.7031 - lr: 0.0010 Epoch 9/20 237/237 [==============================] - 9s 39ms/step - loss: 0.4881 - accuracy: 0.8000 - val_loss: 0.9542 - val_accuracy: 0.6406 - lr: 0.0010 Epoch 10/20 237/237 [==============================] - 9s 40ms/step - loss: 0.4597 - accuracy: 0.8118 - val_loss: 0.9868 - val_accuracy: 0.6484 - lr: 0.0010 Epoch 11/20 237/237 [==============================] - 8s 35ms/step - loss: 0.3974 - accuracy: 0.8425 - val_loss: 0.8226 - val_accuracy: 0.6875 - lr: 0.0010 Epoch 12/20 237/237 [==============================] - 10s 40ms/step - loss: 0.3473 - accuracy: 0.8656 - val_loss: 0.9310 - val_accuracy: 0.6641 - lr: 0.0010 Epoch 13/20 237/237 [==============================] - ETA: 0s - loss: 0.3056 - accuracy: 0.8805 Epoch 13: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026. 237/237 [==============================] - 10s 41ms/step - loss: 0.3056 - accuracy: 0.8805 - val_loss: 1.0509 - val_accuracy: 0.7266 - lr: 0.0010 Epoch 14/20 237/237 [==============================] - 8s 36ms/step - loss: 0.1587 - accuracy: 0.9418 - val_loss: 0.9721 - val_accuracy: 0.7578 - lr: 2.0000e-04 Epoch 15/20 237/237 [==============================] - 9s 40ms/step - loss: 0.0894 - accuracy: 0.9698 - val_loss: 1.2126 - val_accuracy: 0.7344 - lr: 2.0000e-04 Epoch 16/20 237/237 [==============================] - 10s 40ms/step - loss: 0.0578 - accuracy: 0.9815 - val_loss: 1.3537 - val_accuracy: 0.7656 - lr: 2.0000e-04 Epoch 16: early stopping
# evaluate vgg_engine_model on test data
cnn_model_3.evaluate(test_generator_gray_2)
1/1 [==============================] - 0s 351ms/step - loss: 0.8662 - accuracy: 0.8203
[0.8662301301956177, 0.8203125]
plot_accuracy(cnn_3_hist)
plot_loss(cnn_3_hist)
# get test images and their classes
test_images, test_classes = next(test_generator_gray_2)
actual_classes = np.argmax(test_classes, axis = -1)
# Make predictions
cnn_3_predictions = cnn_model_3.predict(test_images)
# Convert predictions to class labels
cnn_3_predictions = np.argmax(cnn_3_predictions, axis = -1)
# generate classification report and confussion matrix
metrics_score(actual_classes,cnn_3_predictions )
4/4 [==============================] - 0s 6ms/step
precision recall f1-score support
0 0.87 0.84 0.86 32
1 0.73 0.75 0.74 32
2 0.75 0.75 0.75 32
3 0.94 0.94 0.94 32
accuracy 0.82 128
macro avg 0.82 0.82 0.82 128
weighted avg 0.82 0.82 0.82 128
Observations and Insights:
# generating comparison of number of trained att and achieved accuracy measures for all models
# create a dict with models, their names, history repositories and test data sets
models = {
cnn_model_1 : ['cnn_model_1', cnn_1_hist_gray, test_generator_gray],
cnn_model_1rgb: ['cnn_model_1rgb', cnn_1_hist_rgb, test_generator_rgb],
cnn_model_2: ['cnn_model_2', cnn_2_hist, test_generator_rgb],
vgg_engine_model: ['vgg_engine_model', vgg_hist, test_generator_rgb_2],
resnet_engine_model: ['resnet_engine_model', resnet_hist,test_generator_rgb_2],
effnet_engine_model:['effnet_engine_model', effnet_hist,test_generator_rgb_2],
cnn_model_3:['cnn_model_3', cnn_3_hist, test_generator_gray_2],
}
titles = [ '-Model Name-', '-Trained Attr-', '-Train Acc-', '-Val Acc-', '-Test Acc-']
print(f'{titles[0]:<24} {titles[1]:<18} {titles[2]:<12} {titles[3]:<12} {titles[4]}')
# run a loop through models and generate required stats for comparison
for model, values in models.items():
print(f'{values[0]:<28}', end='')
print (f'{sum([tf.size(w) for w in model.trainable_variables]).numpy():<19}', end='')
acc = round(values[1].history['accuracy'][-1],2)
print (f'{acc:<13}', end = '')
val_acc = round(values[1].history['val_accuracy'][-1],2)
print (f'{val_acc:<13}', end = '')
loss, accuracy = model.evaluate(values[2], verbose = 0)
print (round(accuracy,2))
-Model Name- -Trained Attr- -Train Acc- -Val Acc- -Test Acc- cnn_model_1 71508 0.79 0.77 0.75 cnn_model_1rgb 72660 0.79 0.65 0.73 cnn_model_2 2668980 0.98 0.72 0.78 vgg_engine_model 804500 0.84 0.65 0.7 resnet_engine_model 67220 0.87 0.73 0.73 effnet_engine_model 67220 0.85 0.7 0.7 cnn_model_3 1176820 0.98 0.77 0.82
Observations and Insights:
The proposed model is cnn_model_3, consisting of 5 convolutional blocks (please refer to cnn_model_3 building section above for detailed specification of the model). This model has performed best of all models across all data set (training, validation, test) , achieving best recall and precision scores on the test set. It outperformed all other models, specially in the classification of neutral and sad facial emotions.
Recommendations for implementation
Before implementing the solution: